2021-08-21T14:28:10.899 INFO:root:teuthology version: 1.1.0-321319b1 2021-08-21T14:28:10.901 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --description rados/standalone/{supported-random-distro$/{centos_8} workloads/osd-backfill} --name yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi --verbose --archive /home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350372 -- /home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350372/orig.config.yaml 2021-08-21T14:28:10.926 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-08-21T14:28:10.975 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350372 branch: wip-yuri-master-8.20.21 description: rados/standalone/{supported-random-distro$/{centos_8} workloads/osd-backfill} email: ceph-qa@ceph.io first_in_suite: false job_id: '6350372' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.3' overrides: admin_socket: branch: wip-yuri-master-8.20.21 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-yuri-master-8.20.21 sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 owner: scheduled_yuriw@teuthology priority: 51 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 sleep_before_teardown: 0 suite: rados suite_branch: wip-yuri-master-8.20.21 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 targets: smithi077.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4QfXReBF2Z2KCbU+WJ4eAwqvhW1CgO9VbYNbTBCgC4YG3nSOanvmjQLmfTin+mJh0hAHUxF8uDUYt7zLPxPqloqc0bdXDPjWiWXRjw8KsIlDPi2ZPNv/vJfH8gLild96JliDm0p252f9FqrG/uNugqU42KX7ML4masiHYmnN+wqFqNOhMEF8k1PRyoZu4zLS9rEvo3aqDD6Ld02xXc4M3D2SKclQR0nYuMoMahhiXmI5V+DWqxqkD+R2/3zrYXnQPZCd4ic/8wEEPpEhkyqFVUjehyy10ApsPW0amhsyIbhp4TUlayYsuZ0ux6MMb6jvrkbaD5C7zWC9wU6J/0sQx8fano+XSj16wwtpWhnZUj0tBl9yo9gcLpOJTWe3BBPl/xcQsGWLYNC/4yfGNCVqWtq7rQwUSDrvnFQmVFzzEyHR/SnVi7ejhAz2Yz4MKLCMjtivv+UomNbsXeNbxIBARtw3vk7DvHgPpfsTErD5OWmfcjo7JZtVw+PvxOTBjjyM= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd-backfill teuthology_branch: master teuthology_sha1: 321319b12ea4ff9b63c7655015a3156de2c3b279 timestamp: 2021-08-21_14:21:15 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.30581 2021-08-21T14:28:10.975 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa; will attempt to use it 2021-08-21T14:28:10.976 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_cae1ffd55a16af4627c36508fde05bf24312fad6/qa/tasks 2021-08-21T14:28:10.976 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-08-21T14:28:10.978 INFO:teuthology.task.internal:Checking packages... 2021-08-21T14:28:10.999 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'cae1ffd55a16af4627c36508fde05bf24312fad6' 2021-08-21T14:28:11.000 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-08-21T14:28:11.000 INFO:teuthology.packaging:ref: None 2021-08-21T14:28:11.001 INFO:teuthology.packaging:tag: None 2021-08-21T14:28:11.001 INFO:teuthology.packaging:branch: wip-yuri-master-8.20.21 2021-08-21T14:28:11.001 INFO:teuthology.packaging:sha1: cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:28:11.002 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-master-8.20.21 2021-08-21T14:28:11.166 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-7135.gcae1ffd5 2021-08-21T14:28:11.166 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-08-21T14:28:11.174 INFO:teuthology.task.internal:no buildpackages task found 2021-08-21T14:28:11.175 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-08-21T14:28:11.204 INFO:teuthology.task.internal:Saving configuration 2021-08-21T14:28:11.213 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-08-21T14:28:11.255 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-08-21T14:28:11.271 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-08-21 14:23:16.991656', 'description': '/home/teuthworker/archive/yuriw-2021-08-21_14:21:15-rados-wip-yuri-master-8.20.21-distro-basic-smithi/6350372', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMKaI+S7nLVtEormG0+EEaX1+TuQ3rCUinPfshTk86dUkyTwIMdDfc7tmX6FqxoGySjFSg7mpU++UUsC28YWOT8=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi077.front.sepia.ceph.com'} 2021-08-21T14:28:11.271 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-08-21T14:28:11.305 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2021-08-21T14:28:11.305 INFO:teuthology.run_tasks:Running task console_log... 2021-08-21T14:28:11.370 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9217161b70>, signals=[15]) 2021-08-21T14:28:11.372 INFO:teuthology.run_tasks:Running task internal.connect... 2021-08-21T14:28:11.407 INFO:teuthology.task.internal:Opening connections... 2021-08-21T14:28:11.407 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:28:11.409 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:28:11.478 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-08-21T14:28:11.488 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2021-08-21T14:28:11.626 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2021-08-21T14:28:11.626 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2021-08-21T14:28:11.680 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="CentOS Linux" 2021-08-21T14:28:11.680 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="8" 2021-08-21T14:28:11.681 INFO:teuthology.orchestra.run.smithi077.stdout:ID="centos" 2021-08-21T14:28:11.681 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE="rhel fedora" 2021-08-21T14:28:11.681 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="8" 2021-08-21T14:28:11.681 INFO:teuthology.orchestra.run.smithi077.stdout:PLATFORM_ID="platform:el8" 2021-08-21T14:28:11.681 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="CentOS Linux 8" 2021-08-21T14:28:11.682 INFO:teuthology.orchestra.run.smithi077.stdout:ANSI_COLOR="0;31" 2021-08-21T14:28:11.682 INFO:teuthology.orchestra.run.smithi077.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-08-21T14:28:11.682 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://centos.org/" 2021-08-21T14:28:11.682 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-08-21T14:28:11.683 INFO:teuthology.orchestra.run.smithi077.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-08-21T14:28:11.683 INFO:teuthology.orchestra.run.smithi077.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-08-21T14:28:11.684 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2021-08-21T14:28:11.709 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-08-21T14:28:11.722 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-08-21T14:28:11.775 INFO:teuthology.task.internal:Checking for old test directory... 2021-08-21T14:28:11.775 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2021-08-21T14:28:11.789 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-08-21T14:28:11.833 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-08-21T14:28:11.833 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2021-08-21T14:28:11.861 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-08-21T14:28:11.900 INFO:teuthology.run_tasks:Running task kernel... 2021-08-21T14:28:11.959 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2021-08-21T14:28:11.959 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-08-21T14:28:11.960 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2021-08-21T14:28:11.974 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-305.10.2.el8_4.x86_64 2021-08-21T14:28:11.975 INFO:teuthology.task.kernel:Running kernel on smithi077: 4.18.0-305.10.2.el8_4.x86_64 2021-08-21T14:28:11.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum install -y kernel 2021-08-21T14:28:14.086 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - AppStream 40 MB/s | 8.8 MB 00:00 2021-08-21T14:28:15.563 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - Base 45 MB/s | 5.6 MB 00:00 2021-08-21T14:28:16.397 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - Extras 160 kB/s | 10 kB 00:00 2021-08-21T14:28:18.327 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Linux 8 - AppStream 4.7 MB/s | 8.8 MB 00:01 2021-08-21T14:28:21.815 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Linux 8 - BaseOS 2.6 MB/s | 5.6 MB 00:02 2021-08-21T14:28:22.735 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Linux 8 - Extras 68 kB/s | 10 kB 00:00 2021-08-21T14:28:22.877 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - PowerTools 26 MB/s | 2.3 MB 00:00 2021-08-21T14:28:25.469 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2021-08-21T14:28:26.482 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 11 MB/s | 10 MB 00:00 2021-08-21T14:28:28.654 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2021-08-21T14:28:30.616 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-305.10.2.el8_4.x86_64 is already installed. 2021-08-21T14:28:30.616 INFO:teuthology.orchestra.run.smithi077.stdout:Package kernel-4.18.0-240.el8.x86_64 is already installed. 2021-08-21T14:28:30.924 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:28:30.925 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:28:30.925 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:28:30.925 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:28:30.926 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:28:30.926 INFO:teuthology.orchestra.run.smithi077.stdout: kernel x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 5.9 M 2021-08-21T14:28:30.926 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:28:30.926 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 36 M 2021-08-21T14:28:30.926 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules x86_64 4.18.0-305.12.1.el8_4 CentOS-Base 28 M 2021-08-21T14:28:30.927 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:28:30.927 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:28:30.927 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:28:30.927 INFO:teuthology.orchestra.run.smithi077.stdout:Install 3 Packages 2021-08-21T14:28:30.928 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:28:30.928 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 70 M 2021-08-21T14:28:30.928 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 88 M 2021-08-21T14:28:30.928 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:28:31.070 INFO:teuthology.orchestra.run.smithi077.stdout:(1/3): kernel-4.18.0-305.12.1.el8_4.x86_64.rpm 41 MB/s | 5.9 MB 00:00 2021-08-21T14:28:31.807 INFO:teuthology.orchestra.run.smithi077.stdout:(2/3): kernel-modules-4.18.0-305.12.1.el8_4.x86 32 MB/s | 28 MB 00:00 2021-08-21T14:28:32.049 INFO:teuthology.orchestra.run.smithi077.stdout:(3/3): kernel-core-4.18.0-305.12.1.el8_4.x86_64 32 MB/s | 36 MB 00:01 2021-08-21T14:28:32.053 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:28:32.053 INFO:teuthology.orchestra.run.smithi077.stdout:Total 62 MB/s | 70 MB 00:01 2021-08-21T14:28:32.165 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:28:32.496 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:28:32.496 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:28:33.688 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:28:33.710 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:28:36.154 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:28:36.659 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-08-21T14:28:39.894 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-08-21T14:28:40.258 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-08-21T14:28:43.234 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-08-21T14:28:43.427 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-08-21T14:29:03.833 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-core-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-08-21T14:29:04.160 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: kernel-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-08-21T14:29:04.161 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-4.18.0-305.12.1.el8_4.x86_64 1/3 2021-08-21T14:29:04.161 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-core-4.18.0-305.12.1.el8_4.x86_64 2/3 2021-08-21T14:29:04.551 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : kernel-modules-4.18.0-305.12.1.el8_4.x86_64 3/3 2021-08-21T14:29:04.552 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:29:04.552 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:29:04.552 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.552 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-core-4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.553 INFO:teuthology.orchestra.run.smithi077.stdout: kernel-modules-4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.553 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:29:04.553 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:29:04.616 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel --last | head -n 1 2021-08-21T14:29:04.766 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Sat 21 Aug 2021 02:28:43 PM UTC 2021-08-21T14:29:04.767 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.768 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.10.2.el8_4.x86_64 Expected: 4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.768 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-08-21T14:29:04.768 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-08-21T14:29:04.769 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-08-21T14:29:04.769 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2021-08-21T14:29:04.769 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q kernel --last | head -n 1 2021-08-21T14:29:04.901 INFO:teuthology.orchestra.run.smithi077.stdout:kernel-4.18.0-305.12.1.el8_4.x86_64 Sat 21 Aug 2021 02:28:43 PM UTC 2021-08-21T14:29:04.903 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.903 DEBUG:teuthology.orchestra.run.smithi077:> sudo rpm -qi grub2-tools 2021-08-21T14:29:04.933 INFO:teuthology.orchestra.run.smithi077.stdout:Name : grub2-tools 2021-08-21T14:29:04.933 INFO:teuthology.orchestra.run.smithi077.stdout:Epoch : 1 2021-08-21T14:29:04.933 INFO:teuthology.orchestra.run.smithi077.stdout:Version : 2.02 2021-08-21T14:29:04.934 INFO:teuthology.orchestra.run.smithi077.stdout:Release : 90.el8 2021-08-21T14:29:04.934 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: x86_64 2021-08-21T14:29:04.934 INFO:teuthology.orchestra.run.smithi077.stdout:Install Date: Wed 11 Aug 2021 06:38:04 PM UTC 2021-08-21T14:29:04.934 INFO:teuthology.orchestra.run.smithi077.stdout:Group : System Environment/Base 2021-08-21T14:29:04.935 INFO:teuthology.orchestra.run.smithi077.stdout:Size : 9197207 2021-08-21T14:29:04.935 INFO:teuthology.orchestra.run.smithi077.stdout:License : GPLv3+ 2021-08-21T14:29:04.935 INFO:teuthology.orchestra.run.smithi077.stdout:Signature : RSA/SHA256, Tue 08 Sep 2020 07:14:39 PM UTC, Key ID 05b555b38483c65d 2021-08-21T14:29:04.935 INFO:teuthology.orchestra.run.smithi077.stdout:Source RPM : grub2-2.02-90.el8.src.rpm 2021-08-21T14:29:04.936 INFO:teuthology.orchestra.run.smithi077.stdout:Build Date : Tue 08 Sep 2020 07:00:29 PM UTC 2021-08-21T14:29:04.936 INFO:teuthology.orchestra.run.smithi077.stdout:Build Host : kbuilder.bsys.centos.org 2021-08-21T14:29:04.936 INFO:teuthology.orchestra.run.smithi077.stdout:Relocations : (not relocatable) 2021-08-21T14:29:04.936 INFO:teuthology.orchestra.run.smithi077.stdout:Packager : CentOS BuildSystem 2021-08-21T14:29:04.936 INFO:teuthology.orchestra.run.smithi077.stdout:Vendor : CentOS 2021-08-21T14:29:04.937 INFO:teuthology.orchestra.run.smithi077.stdout:URL : http://www.gnu.org/software/grub/ 2021-08-21T14:29:04.937 INFO:teuthology.orchestra.run.smithi077.stdout:Summary : Support tools for GRUB. 2021-08-21T14:29:04.937 INFO:teuthology.orchestra.run.smithi077.stdout:Description : 2021-08-21T14:29:04.937 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:29:04.938 INFO:teuthology.orchestra.run.smithi077.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-08-21T14:29:04.938 INFO:teuthology.orchestra.run.smithi077.stdout:customizable bootloader with modular architecture. It supports a rich 2021-08-21T14:29:04.938 INFO:teuthology.orchestra.run.smithi077.stdout:variety of kernel formats, file systems, computer architectures and 2021-08-21T14:29:04.938 INFO:teuthology.orchestra.run.smithi077.stdout:hardware devices. 2021-08-21T14:29:04.938 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:29:04.939 INFO:teuthology.orchestra.run.smithi077.stdout:This subpackage provides tools for support of all platforms. 2021-08-21T14:29:04.940 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-08-21T14:29:04.940 INFO:teuthology.task.kernel:Updating grub on smithi077 to boot 4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:04.940 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-08-21T14:29:06.614 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2021-08-21T14:29:08.905 INFO:teuthology.orchestra.run.smithi077.stderr:done 2021-08-21T14:29:08.907 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2021-08-21T14:29:08.951 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.pDnpdJfW1C 2021-08-21T14:29:08.951 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pDnpdJfW1C 2021-08-21T14:29:09.021 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.pDnpdJfW1C 2021-08-21T14:29:09.180 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.pDnpdJfW1C is 6KB 2021-08-21T14:29:09.229 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.pDnpdJfW1C 2021-08-21T14:29:09.243 DEBUG:teuthology.orchestra.run.smithi077:> sudo /bin/ls /boot/loader/entries || true 2021-08-21T14:29:09.318 INFO:teuthology.orchestra.run.smithi077.stdout:ecc66dffcd264e5181894e69c53ff13c-0-rescue.conf 2021-08-21T14:29:09.319 INFO:teuthology.orchestra.run.smithi077.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-240.el8.x86_64.conf 2021-08-21T14:29:09.319 INFO:teuthology.orchestra.run.smithi077.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.10.2.el8_4.x86_64.conf 2021-08-21T14:29:09.320 INFO:teuthology.orchestra.run.smithi077.stdout:ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64.conf 2021-08-21T14:29:09.321 DEBUG:teuthology.orchestra.run.smithi077:> sudo grub2-set-default ecc66dffcd264e5181894e69c53ff13c-4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:29:09.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2021-08-21T14:29:39.522 INFO:teuthology.misc:Re-opening connections... 2021-08-21T14:29:39.523 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:29:39.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:29:39.525 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:29:58.095 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2021-08-21T14:29:58.096 DEBUG:teuthology.misc:waited 18.57222557067871 2021-08-21T14:29:59.097 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:29:59.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:29:59.098 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:30:02.155 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2021-08-21T14:30:02.155 DEBUG:teuthology.misc:waited 22.63178586959839 2021-08-21T14:30:03.157 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:30:03.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:30:03.158 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:30:05.230 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2021-08-21T14:30:05.231 DEBUG:teuthology.misc:waited 25.707736492156982 2021-08-21T14:30:06.232 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:30:06.233 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:30:06.233 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:30:08.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2021-08-21T14:30:08.299 DEBUG:teuthology.misc:waited 28.77565360069275 2021-08-21T14:30:09.300 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:30:09.301 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:30:09.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:30:11.370 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2021-08-21T14:30:11.371 DEBUG:teuthology.misc:waited 31.847681522369385 2021-08-21T14:30:12.373 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:30:12.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:30:12.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:31:12.435 DEBUG:teuthology.orchestra.remote:timed out 2021-08-21T14:31:12.436 DEBUG:teuthology.misc:waited 92.91283464431763 2021-08-21T14:31:13.438 INFO:teuthology.misc:trying to connect to ubuntu@smithi077.front.sepia.ceph.com 2021-08-21T14:31:13.438 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:31:13.439 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:31:13.752 DEBUG:teuthology.orchestra.run.smithi077:> true 2021-08-21T14:31:14.048 DEBUG:teuthology.misc:waited 94.52428603172302 2021-08-21T14:31:15.049 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-08-21T14:31:15.050 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-305.12.1.el8_4.x86_64"... 2021-08-21T14:31:15.050 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2021-08-21T14:31:15.067 INFO:teuthology.orchestra.run.smithi077.stdout:4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:31:15.067 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-305.12.1.el8_4.x86_64 vs 4.18.0-305.12.1.el8_4.x86_64 2021-08-21T14:31:15.068 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-08-21T14:31:15.068 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-08-21T14:31:16.069 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-08-21T14:31:16.070 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-08-21T14:31:16.146 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2021-08-21T14:31:16.162 INFO:teuthology.run_tasks:Running task internal.base... 2021-08-21T14:31:16.174 INFO:teuthology.task.internal:Creating test directory... 2021-08-21T14:31:16.175 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-08-21T14:31:16.191 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-08-21T14:31:16.211 INFO:teuthology.run_tasks:Running task internal.archive... 2021-08-21T14:31:16.371 INFO:teuthology.task.internal:Creating archive directory... 2021-08-21T14:31:16.371 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-08-21T14:31:16.403 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-08-21T14:31:16.494 INFO:teuthology.task.internal:Enabling coredump saving... 2021-08-21T14:31:16.495 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-08-21T14:31:16.526 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:31:16.537 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:31:16.539 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-08-21T14:31:16.619 INFO:teuthology.task.internal:Configuring sudo... 2021-08-21T14:31:16.620 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-08-21T14:31:16.649 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-08-21T14:31:16.727 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-08-21T14:31:16.727 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-08-21T14:31:16.745 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-08-21T14:31:16.822 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-08-21T14:31:16.889 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-08-21T14:31:16.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-08-21T14:31:16.974 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-08-21T14:31:16.975 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-08-21T14:31:17.041 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2021-08-21T14:31:17.141 INFO:teuthology.orchestra.run.smithi077.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-08-21T14:31:17.158 INFO:teuthology.run_tasks:Running task internal.timer... 2021-08-21T14:31:17.170 INFO:teuthology.task.internal:Starting timer... 2021-08-21T14:31:17.170 INFO:teuthology.run_tasks:Running task pcp... 2021-08-21T14:31:17.223 INFO:teuthology.run_tasks:Running task selinux... 2021-08-21T14:31:17.308 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-08-21T14:31:17.309 DEBUG:teuthology.orchestra.run.smithi077:> sudo service auditd rotate 2021-08-21T14:31:17.371 INFO:teuthology.orchestra.run.smithi077.stdout:Rotating logs: [ OK ] 2021-08-21T14:31:17.373 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-08-21T14:31:17.373 DEBUG:teuthology.orchestra.run.smithi077:> /usr/sbin/getenforce 2021-08-21T14:31:17.410 INFO:teuthology.orchestra.run.smithi077.stdout:Permissive 2021-08-21T14:31:17.410 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi077.front.sepia.ceph.com': 'permissive'} 2021-08-21T14:31:17.411 DEBUG:teuthology.orchestra.run.smithi077:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-08-21T14:31:17.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-08-21T14:31:17.475 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-08-21T14:31:17.476 DEBUG:teuthology.orchestra.run.smithi077:> sudo /usr/sbin/setenforce permissive 2021-08-21T14:31:17.542 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-08-21T14:31:17.554 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-08-21T14:31:17.555 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-08-21T14:31:17.574 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2021-08-21T14:31:17.576 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-08-21T14:31:17.954 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-08-21T14:31:18.791 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-08-21T14:31:21.751 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-08-21T14:31:21.775 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:21 +0000 (0:00:00.041) 0:00:00.041 ******* 2021-08-21T14:31:24.331 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi077.front.sepia.ceph.com] 2021-08-21T14:31:24.380 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:24 +0000 (0:00:02.605) 0:00:02.647 ******* 2021-08-21T14:31:24.646 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-08-21T14:31:24.648 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-08-21T14:31:24.670 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:24 +0000 (0:00:00.289) 0:00:02.937 ******* 2021-08-21T14:31:25.049 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-08-21T14:31:25.051 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-08-21T14:31:25.074 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:25 +0000 (0:00:00.404) 0:00:03.341 ******* 2021-08-21T14:31:25.133 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-08-21T14:31:25.135 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:31:25.155 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:25 +0000 (0:00:00.081) 0:00:03.422 ******* 2021-08-21T14:31:25.421 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:31:25.443 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:25 +0000 (0:00:00.288) 0:00:03.710 ******* 2021-08-21T14:31:26.356 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-08-21T14:31:26.358 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1629556285.4968514-16577-10785062588074/source", "state": "file", "uid": 0} 2021-08-21T14:31:26.382 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:26 +0000 (0:00:00.938) 0:00:04.648 ******* 2021-08-21T14:31:26.807 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-08-21T14:31:26.809 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-08-21T14:31:26.811 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-08-21T14:31:26.883 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2021-08-21T14:31:26.885 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY [testnodes] *************************************************************** 2021-08-21T14:31:26.910 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-08-21T14:31:26.912 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:26 +0000 (0:00:00.528) 0:00:05.177 ******* 2021-08-21T14:31:27.443 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] 2021-08-21T14:31:27.460 INFO:teuthology.task.ansible.out: TASK [set_fact] **************************************************************** Saturday 21 August 2021 14:31:27 +0000 (0:00:00.549) 0:00:05.726 ******* 2021-08-21T14:31:27.489 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"ran_from_cephlab_playbook": true}, "changed": false} 2021-08-21T14:31:27.493 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-08-21T14:31:27.654 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:27 +0000 (0:00:00.193) 0:00:05.920 ******* 2021-08-21T14:31:28.288 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-08-21T14:31:28.290 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] 2021-08-21T14:31:28.350 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Saturday 21 August 2021 14:31:28 +0000 (0:00:00.695) 0:00:06.616 ******* 2021-08-21T14:31:29.111 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-08-21T14:31:29.112 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.276668", "end": "2021-08-21 14:31:29.079587", "rc": 0, "start": "2021-08-21 14:31:28.802919", "stderr": "", "stderr_lines": [], "stdout": "7f6ec18127eb11437e244122daa7109a4352270a", "stdout_lines": ["7f6ec18127eb11437e244122daa7109a4352270a"]} 2021-08-21T14:31:29.132 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.782) 0:00:07.399 ******* 2021-08-21T14:31:29.325 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.016843", "end": "2021-08-21 14:31:29.309325", "failed_when_result": false, "rc": 0, "start": "2021-08-21 14:31:29.292482", "stderr": "", "stderr_lines": [], "stdout": "c29274177b79f72740fc714fa28b55400cd0737f", "stdout_lines": ["c29274177b79f72740fc714fa28b55400cd0737f"]} 2021-08-21T14:31:29.348 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.215) 0:00:07.614 ******* 2021-08-21T14:31:29.417 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-08-21T14:31:29.418 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:31:29.439 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.090) 0:00:07.705 ******* 2021-08-21T14:31:29.508 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:31:29.529 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.090) 0:00:07.796 ******* 2021-08-21T14:31:29.601 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-08-21T14:31:29.603 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:31:29.626 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.096) 0:00:07.892 ******* 2021-08-21T14:31:29.697 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-08-21T14:31:29.700 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:31:29.720 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.094) 0:00:07.987 ******* 2021-08-21T14:31:29.801 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-08-21T14:31:29.802 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:31:29.823 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:29 +0000 (0:00:00.102) 0:00:08.089 ******* 2021-08-21T14:31:30.059 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:31:30.082 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:31:30 +0000 (0:00:00.259) 0:00:08.349 ******* 2021-08-21T14:31:30.364 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-08-21T14:31:30.366 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-08-21T14:31:30.565 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-08-21T14:31:30.755 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-08-21T14:31:30.954 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-08-21T14:31:31.145 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-08-21T14:31:31.349 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-08-21T14:31:31.541 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-08-21T14:31:31.735 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-08-21T14:31:31.944 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-08-21T14:31:32.133 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-08-21T14:31:32.337 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-08-21T14:31:32.532 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-08-21T14:31:32.731 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-08-21T14:31:32.931 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-08-21T14:31:33.129 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-08-21T14:31:33.319 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-08-21T14:31:33.517 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-08-21T14:31:33.708 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-08-21T14:31:33.908 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-08-21T14:31:34.110 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-08-21T14:31:34.299 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-08-21T14:31:34.498 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-08-21T14:31:34.698 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-08-21T14:31:34.897 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-08-21T14:31:35.087 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-08-21T14:31:35.289 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-08-21T14:31:35.480 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-08-21T14:31:35.678 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-08-21T14:31:35.868 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-08-21T14:31:36.070 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-08-21T14:31:36.270 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-08-21T14:31:36.459 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-08-21T14:31:36.668 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-08-21T14:31:36.867 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-08-21T14:31:37.063 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-08-21T14:31:37.261 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-08-21T14:31:37.459 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-08-21T14:31:37.656 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-08-21T14:31:37.848 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-08-21T14:31:38.045 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-08-21T14:31:38.244 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-08-21T14:31:38.435 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-08-21T14:31:38.634 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-08-21T14:31:38.827 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-08-21T14:31:39.026 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-08-21T14:31:39.230 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-08-21T14:31:39.430 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-08-21T14:31:39.620 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-08-21T14:31:39.822 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-08-21T14:31:40.022 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-08-21T14:31:40.213 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-08-21T14:31:40.404 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-08-21T14:31:40.606 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-08-21T14:31:40.795 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-08-21T14:31:40.996 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-08-21T14:31:41.196 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-08-21T14:31:41.384 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-08-21T14:31:41.595 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-08-21T14:31:41.781 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-08-21T14:31:41.993 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-08-21T14:31:42.183 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-08-21T14:31:42.382 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-08-21T14:31:42.582 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-08-21T14:31:42.776 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-08-21T14:31:42.966 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-08-21T14:31:43.169 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-08-21T14:31:43.357 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-08-21T14:31:43.560 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-08-21T14:31:43.755 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-08-21T14:31:43.943 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-08-21T14:31:44.144 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-08-21T14:31:44.334 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-08-21T14:31:44.537 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-08-21T14:31:44.731 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-08-21T14:31:44.932 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-08-21T14:31:45.132 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-08-21T14:31:45.321 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-08-21T14:31:45.522 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-08-21T14:31:45.715 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-08-21T14:31:45.925 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-08-21T14:31:46.114 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-08-21T14:31:46.304 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-08-21T14:31:46.505 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-08-21T14:31:46.711 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-08-21T14:31:46.909 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-08-21T14:31:47.115 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-08-21T14:31:47.306 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-08-21T14:31:47.504 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-08-21T14:31:47.716 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-08-21T14:31:47.893 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-08-21T14:31:48.092 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-08-21T14:31:48.293 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-08-21T14:31:48.492 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-08-21T14:31:48.681 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-08-21T14:31:48.878 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-08-21T14:31:49.068 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-08-21T14:31:49.267 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-08-21T14:31:49.458 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-08-21T14:31:49.659 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-08-21T14:31:49.860 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-08-21T14:31:50.048 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-08-21T14:31:50.247 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-08-21T14:31:50.440 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-08-21T14:31:50.640 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-08-21T14:31:50.841 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-08-21T14:31:51.030 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-08-21T14:31:51.219 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-08-21T14:31:51.423 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-08-21T14:31:51.613 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-08-21T14:31:51.808 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-08-21T14:31:52.014 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-08-21T14:31:52.225 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-08-21T14:31:52.424 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-08-21T14:31:52.625 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-08-21T14:31:52.826 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-08-21T14:31:53.016 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-08-21T14:31:53.217 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-08-21T14:31:53.418 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-08-21T14:31:53.612 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-08-21T14:31:53.815 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-08-21T14:31:54.008 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-08-21T14:31:54.210 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-08-21T14:31:54.444 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-08-21T14:31:54.638 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-08-21T14:31:54.845 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-08-21T14:31:55.039 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-08-21T14:31:55.241 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-08-21T14:31:55.443 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-08-21T14:31:55.638 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-08-21T14:31:55.839 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-08-21T14:31:56.062 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-08-21T14:31:56.254 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-08-21T14:31:56.453 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-08-21T14:31:56.656 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-08-21T14:31:56.866 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-08-21T14:31:57.066 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-08-21T14:31:57.263 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-08-21T14:31:57.466 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-08-21T14:31:57.658 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/dachary", "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "move_home": false, "name": "dachary", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-08-21T14:31:57.871 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1238, "groups": "sudo", "home": "/home/amctagga", "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "move_home": false, "name": "amctagga", "shell": "/bin/bash", "state": "present", "uid": 1238} 2021-08-21T14:31:58.071 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1239, "groups": "sudo", "home": "/home/kevinz", "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "move_home": false, "name": "kevinz", "shell": "/bin/bash", "state": "present", "uid": 1239} 2021-08-21T14:31:58.262 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1240, "groups": "sudo", "home": "/home/liu-chunmei", "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "move_home": false, "name": "liu-chunmei", "shell": "/bin/bash", "state": "present", "uid": 1240} 2021-08-21T14:31:58.471 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1241, "groups": "sudo", "home": "/home/cyx1231st", "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "move_home": false, "name": "cyx1231st", "shell": "/bin/bash", "state": "present", "uid": 1241} 2021-08-21T14:31:58.669 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1242, "groups": "sudo", "home": "/home/sunnagar", "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "move_home": false, "name": "sunnagar", "shell": "/bin/bash", "state": "present", "uid": 1242} 2021-08-21T14:31:58.881 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1243, "groups": "sudo", "home": "/home/lflores", "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "move_home": false, "name": "lflores", "shell": "/bin/bash", "state": "present", "uid": 1243} 2021-08-21T14:31:59.066 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1244, "groups": "sudo", "home": "/home/jsawaya", "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "move_home": false, "name": "jsawaya", "shell": "/bin/bash", "state": "present", "uid": 1244} 2021-08-21T14:31:59.260 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1245, "groups": "sudo", "home": "/home/markw8973240", "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "move_home": false, "name": "markw8973240", "shell": "/bin/bash", "state": "present", "uid": 1245} 2021-08-21T14:31:59.474 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1246, "groups": "sudo", "home": "/home/luisrico", "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "move_home": false, "name": "luisrico", "shell": "/bin/bash", "state": "present", "uid": 1246} 2021-08-21T14:31:59.660 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1247, "groups": "sudo", "home": "/home/matan", "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "move_home": false, "name": "matan", "shell": "/bin/bash", "state": "present", "uid": 1247} 2021-08-21T14:31:59.863 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1248, "groups": "sudo", "home": "/home/ozeneva", "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "move_home": false, "name": "ozeneva", "shell": "/bin/bash", "state": "present", "uid": 1248} 2021-08-21T14:32:00.064 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1249, "groups": "sudo", "home": "/home/sudhanaruna", "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "move_home": false, "name": "sudhanaruna", "shell": "/bin/bash", "state": "present", "uid": 1249} 2021-08-21T14:32:00.263 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1250, "groups": "sudo", "home": "/home/zl31wang", "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "move_home": false, "name": "zl31wang", "shell": "/bin/bash", "state": "present", "uid": 1250} 2021-08-21T14:32:00.464 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1251, "groups": "sudo", "home": "/home/mayagilad", "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "move_home": false, "name": "mayagilad", "shell": "/bin/bash", "state": "present", "uid": 1251} 2021-08-21T14:32:00.654 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1252, "groups": "sudo", "home": "/home/arjsharm", "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "move_home": false, "name": "arjsharm", "shell": "/bin/bash", "state": "present", "uid": 1252} 2021-08-21T14:32:00.867 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1253, "groups": "sudo", "home": "/home/melissali", "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "move_home": false, "name": "melissali", "shell": "/bin/bash", "state": "present", "uid": 1253} 2021-08-21T14:32:02.513 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/ionutbalutoiu", "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "name": "ionutbalutoiu", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-08-21T14:32:04.190 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/pdiazbou", "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "name": "pdiazbou", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-08-21T14:32:05.707 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/xuwang", "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "name": "xuwang", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-08-21T14:32:05.871 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:05 +0000 (0:00:35.788) 0:00:44.137 ******* 2021-08-21T14:32:05.950 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-08-21T14:32:05.972 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:05 +0000 (0:00:00.101) 0:00:44.239 ******* 2021-08-21T14:32:06.066 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-08-21T14:32:06.069 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-08-21T14:32:06.071 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-08-21T14:32:06.073 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-08-21T14:32:06.074 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-08-21T14:32:06.076 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9 2021-08-21T14:32:06.078 INFO:teuthology.task.ansible.out:b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", " 2021-08-21T14:32:06.080 INFO:teuthology.task.ansible.out:ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn" 2021-08-21T14:32:06.081 INFO:teuthology.task.ansible.out:: "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 2021-08-21T14:32:06.083 INFO:teuthology.task.ansible.out:f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": 2021-08-21T14:32:06.084 INFO:teuthology.task.ansible.out:"alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon 2021-08-21T14:32:06.087 INFO:teuthology.task.ansible.out:", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"nam 2021-08-21T14:32:06.089 INFO:teuthology.task.ansible.out:e": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e104 2021-08-21T14:32:06.090 INFO:teuthology.task.ansible.out:6655420d4d0a79ce5667bec1b469a0100"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449 2021-08-21T14:32:06.094 INFO:teuthology.task.ansible.out:ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn 2021-08-21T14:32:06.095 INFO:teuthology.task.ansible.out:": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185 2021-08-21T14:32:06.097 INFO:teuthology.task.ansible.out:052887b82ba5c343a42eb271747a"}, {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4" 2021-08-21T14:32:06.099 INFO:teuthology.task.ansible.out:}, {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}]}, "changed": false} 2021-08-21T14:32:06.119 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:06 +0000 (0:00:00.146) 0:00:44.385 ******* 2021-08-21T14:32:07.204 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* ok: [smithi077.front.sepia.ceph.com] => {"after": "7f6ec18127eb11437e244122daa7109a4352270a", "attempts": 1, "before": "7f6ec18127eb11437e244122daa7109a4352270a", "changed": false, "remote_url_changed": false} 2021-08-21T14:32:07.227 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:07 +0000 (0:00:01.108) 0:00:45.493 ******* 2021-08-21T14:32:07.550 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-08-21T14:32:07.552 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-08-21T14:32:07.760 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-08-21T14:32:07.965 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-08-21T14:32:08.176 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-08-21T14:32:08.177 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-08-21T14:32:08.385 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-08-21T14:32:08.588 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-08-21T14:32:08.589 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-08-21T14:32:08.797 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-08-21T14:32:09.000 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-08-21T14:32:09.210 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-08-21T14:32:09.222 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:09.422 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-08-21T14:32:09.424 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-08-21T14:32:09.634 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-08-21T14:32:09.635 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-08-21T14:32:09.645 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:09.657 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:09.856 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-08-21T14:32:10.069 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-08-21T14:32:10.071 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-08-21T14:32:10.271 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-08-21T14:32:10.482 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-08-21T14:32:10.484 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-08-21T14:32:10.692 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-08-21T14:32:10.901 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-08-21T14:32:11.123 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-08-21T14:32:11.345 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-08-21T14:32:11.347 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-08-21T14:32:11.556 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-08-21T14:32:11.558 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-08-21T14:32:11.766 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-08-21T14:32:11.980 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-08-21T14:32:12.200 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlI 2021-08-21T14:32:12.202 INFO:teuthology.task.ansible.out:Qv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrSX8nDL2WaHe0b/rjE3TZhj5VwbrG6nn/hyWmm2dWcUqotSp973yD2G4tQL9UygeSkDGmjb6whrz2mcAFJ5P4oLtIcCLyZC550dcpwV3BtP/cMAIGE6xeEARKiP8KlVGFNDRg9/l1aAd08Bepe+rJyAYD7IxOOBfgKJMKbeOWnT40RTfwiD5CXNwjzQuhzZDEbm+8zjTChFM4P1EkfR3c+0X3MYpA1AsRQY0ElvFtcn6G0yPKR+obGdIs+ydGupKh1+peGWLemHMA5e62Bs4/Lwtyebyp7AOguC3ZBJ3TZWL4Vn1q6z21cQe6AuowjPrneZWC4ifFB6OUQJqM+R00pHBcajcvsfpYOblZvTJHM726Ek1vahOYfcbEMpqWlVYo+WPRUilzl5cI3ReyAcwphebqKPbhmzCuyJw3lWBKIVDhWjEFzGkxV/iFU5rIXFm/Z0JGNFFxmY1U4GlA4lFXtL7OvEdMsiZo1x0cNmFWLu0k9fQBtiQI19ZBiA7FPKc= yehuda@crestview", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-08-21T14:32:12.400 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-08-21T14:32:12.623 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-08-21T14:32:12.825 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-08-21T14:32:13.046 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-08-21T14:32:13.257 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-08-21T14:32:13.459 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-08-21T14:32:13.461 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-08-21T14:32:13.683 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-08-21T14:32:13.884 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-08-21T14:32:14.084 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-08-21T14:32:14.293 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-08-21T14:32:14.495 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-08-21T14:32:14.709 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-08-21T14:32:14.907 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-08-21T14:32:15.115 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-08-21T14:32:15.337 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-08-21T14:32:15.339 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-08-21T14:32:15.546 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-08-21T14:32:15.549 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-08-21T14:32:15.757 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-08-21T14:32:15.961 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-08-21T14:32:16.170 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-08-21T14:32:16.399 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNH 2021-08-21T14:32:16.400 INFO:teuthology.task.ansible.out:ORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-08-21T14:32:16.597 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-08-21T14:32:16.599 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-08-21T14:32:16.601 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-08-21T14:32:16.806 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-08-21T14:32:17.007 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-08-21T14:32:17.209 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-08-21T14:32:17.419 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-08-21T14:32:17.631 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-08-21T14:32:17.832 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-08-21T14:32:17.833 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-08-21T14:32:18.037 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-08-21T14:32:18.248 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-08-21T14:32:18.458 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-08-21T14:32:18.460 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-08-21T14:32:18.668 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-08-21T14:32:18.869 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-08-21T14:32:19.081 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-08-21T14:32:19.082 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-08-21T14:32:19.295 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-08-21T14:32:19.493 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-08-21T14:32:19.704 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-08-21T14:32:19.915 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-08-21T14:32:20.119 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-08-21T14:32:20.327 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-08-21T14:32:20.559 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-08-21T14:32:20.759 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-08-21T14:32:20.972 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-08-21T14:32:21.185 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-08-21T14:32:21.394 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-08-21T14:32:21.595 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-08-21T14:32:21.805 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-08-21T14:32:22.011 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-08-21T14:32:22.223 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-08-21T14:32:22.424 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-08-21T14:32:22.636 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-08-21T14:32:22.839 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-08-21T14:32:23.051 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-08-21T14:32:23.261 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-08-21T14:32:23.486 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-08-21T14:32:23.488 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-08-21T14:32:23.692 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-08-21T14:32:23.902 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-08-21T14:32:24.115 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-08-21T14:32:24.335 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-08-21T14:32:24.546 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-08-21T14:32:24.548 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-08-21T14:32:24.747 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-08-21T14:32:24.749 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-08-21T14:32:24.961 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-08-21T14:32:25.171 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-08-21T14:32:25.372 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-08-21T14:32:25.589 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-08-21T14:32:25.796 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-08-21T14:32:25.996 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-08-21T14:32:26.218 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-08-21T14:32:26.441 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-08-21T14:32:26.442 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-08-21T14:32:26.642 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-08-21T14:32:26.865 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-08-21T14:32:27.074 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-08-21T14:32:27.294 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-08-21T14:32:27.502 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-08-21T14:32:27.713 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-08-21T14:32:27.923 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDISEXvGI2oJLTRR15ePZLkmdYjaQJxt9V/ 2021-08-21T14:32:27.925 INFO:teuthology.task.ansible.out:yBwKdGqYvnPzmLOpdmVpdQJo7ZOWwdp5tJsCy4q5uX5vIUv6Gyi3uVSGmZnrKUbsqoDNwAX5vx/gFBINIs/ZXXyD4fpUT8aoZeq1XmXdIaWG0nImsrtRnRxKP9NGZEcW8z0WPwgF0Ck+RH0qZUix6Bfbgy7vMXnivuDv+NdjEwyszcrPWvf6Krr0am9Q0+9nj/mjvk6CZLl+CH3HNdkhZuNs+qMG4XW8KhLvQB2p790XVlTHuVU4jWrFmjymggZxkhoCY/ukHH6wi2Pbx6yjjKOj8iIw1xKabJzJ7S8UKT23A41d918DUCUuEG66pN6Z4dnv+fZJxj5iEo1GdycSWIc4SpLTK3z4zDZypVKIzjI4o8VwQaN48HkVjUdTICTvGQfytFi3jX1I5nxIZiiPcxMQLR5hQggP8pn2XtvXpomn0aBc8tVuuZJLVq4zCVnhlVB6IACmb71evjFGOsoSuEsaTufUr48= prsrivas@t480.lab.blr.redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-08-21T14:32:28.134 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-08-21T14:32:28.332 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-08-21T14:32:28.543 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-08-21T14:32:28.766 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-08-21T14:32:28.968 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-08-21T14:32:29.180 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-08-21T14:32:29.393 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-08-21T14:32:29.604 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-08-21T14:32:29.806 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-08-21T14:32:30.019 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-08-21T14:32:30.243 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-08-21T14:32:30.454 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-08-21T14:32:30.668 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-08-21T14:32:30.882 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-08-21T14:32:31.085 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-08-21T14:32:31.296 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-08-21T14:32:31.501 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-08-21T14:32:31.719 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-08-21T14:32:31.922 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-08-21T14:32:31.924 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-08-21T14:32:32.159 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-08-21T14:32:32.367 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-08-21T14:32:32.576 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-08-21T14:32:32.780 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-08-21T14:32:32.997 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-08-21T14:32:33.212 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-08-21T14:32:33.419 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-08-21T14:32:33.635 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-08-21T14:32:33.845 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-08-21T14:32:34.046 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-08-21T14:32:34.255 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-08-21T14:32:34.470 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-08-21T14:32:34.682 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-08-21T14:32:34.886 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-08-21T14:32:35.097 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-08-21T14:32:35.309 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJOefD/VJfS15LRyuVDLluKv93Zgocsl8BhiDMimvBNbrGWc795v2+eevWuwaL/SlL44SnsNZ9Mz8FVn9YuFoQiHLg7cHZNLg9EJh5SMXzZzwLZHnF02MdI2CU6U++9nmh8FtBrdBWKzZ7o+01wOw+52dYOxvP7IJ7C/vjwOLgfigEN8Nd1PrxALVNIQz7fsiet0n3odk9PJfoqTu3IloXm9qbQiPFyfMA6b8a20RdLZCJDGGG5rd6gtsyeFHoAPZEz8FEUlNkow7NM7ZhdxOz/fWXyMZ27KmY8WSAgflbxysMzexyifkGzypiE3NQ10hdPGBgoYsHrAN4Qmte1/Z4aI5nxrNzLy/uNc+L/Bp/0XkLj9zfoiFFOReXvlz/laAV6LpbHyacr+iCwv9rlI7echXmvC6mSJXfBPuyR1uromUXpYRhEkn5uiOsCIQCuYpx2jeil7JV+LJlF62lLSndOv4g12Gq2TEZ4Bhsr2902kftPkLFenyfDTh3A2qGGNM= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-08-21T14:32:35.511 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", "key_ 2021-08-21T14:32:35.513 INFO:teuthology.task.ansible.out:options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-08-21T14:32:35.723 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-08-21T14:32:35.944 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC//ZCaEeVN/J09FtsvuZ4eEs4rTBtYoQyG+ZpuWkJLL90VyJeYrZzRsgvveKW2W9oFxCFcMy7IBhLes0TZ/X/4Ox9eUmP/eyqK8fBqT3K1F1W/uOmKLFhlQhkxxj+p9BOMBD5trfES/QJQ6UFSYIS2WhWaitDgONjoLMR6wXZ9r7yZeQBxbmG4wGI0Cwx4Wgg+I59UbE/5iUfnX++gN3rXrPnv9V0yFeciS4HvXRFXdrIzvQFHcZMmpOOmGruHXN/ME2dZCG+kaZhZdI6kICSlOwpU7TU1lv99zjBCeEzGfFkjv041ykKdNVR3qE0k5u0AOMqxJBORL/woSCdGvrBXqju2G/2ohl3B4zD6TAve+JLfWOV5C20NFRPPbTd53JaNpV9OQqwOfVniowzSYYBdedjm9sq6YwhJt8iBXze3OA8ZO0GPXo7PC2MyrrUtu6uu3DZqU4x0cl8iJxqOlx5pH0Gydj8nuUG0+X6wHZzPXZRe5M7/RzmvXch9koG6FQMiikNjUkK8Guz4NMBdAYUpxYkdPGBhiuELoaAcgIVzSGoZTuICpNYwAxBHk6q18G8Qd8vaeUdGdim4YvwycsSOwFE47wg/XmpCHrNJUZzzB9UD7Kn84TG82LAmCuAzpzcp0uPjsBckbXzj9Zt0hRgF80yDgUazsQkbgtd2p8KxIQ== loic", "key_options": null, "keyfile": "/home/dachary/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dachary", "validate_certs": true} 2021-08-21T14:32:36.157 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwVh9KqjV0b+65aBxQV66IUwNo3D5zTbN8U6Kh6eE4QtaIVe5Mmvi2SBbGMSWofE0zIIw2JmsI5LzdOoCyZl/3gNDPHQd6D9o0xRHQv94giZ2a5K+mW79foAxsuppTjV/Fbg8EZ/19W91+BqMWotlMR66ka0RPT/6U8V5mzXoydSminViO/B6q7ZYzIIgGZqaTsSxCMvz5BAzzeEaGfSlGxw5MOye9sF/tDppuRKG4rMUZULEWGuxgXAtabtXCduocAArqAY3BRJEgj7JetPdPNjipGrULyYQmT7GeknvP/waC0sPm5o8t7taJca7Vu9qKKMKPAqiK4KhlrW/yVBFliT2CjAdVuYvGj87LCcEn6NMeF+4ONwDz++YwjrAWQT2zHUJPvLnYOZPiD1LUm3F34cEHD1y4Uf+oeiYgriUMauskfFn+ShbTmWWooFRFyNobzweg9aMBZSkA+kFFifqOV1FwHTOYg342NumRGB8svK0vvTUID7AU1Ev8YXYqCEGbb0UHth/1pEbAF103enLTtPZ6qKua74rE0qJ89OF9dZXv4JY9+8Juyps7bi2XEyHNm1nHZOY5tqdY0eKFmSGJMoWBDzGgdMjmXlWp4jnY2fC619nZqxMrCQpeT70CqnBhtBemUuu6jHYUbbrPjnXIkH3FZ7/qvZt0jF0IMP9FGw== amctagga@redhat.com", "key_options": null, "keyfile": "/home/amctagga/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amctagga", "validate_certs": true} 2021-08-21T14:32:36.359 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDtiHTDuQk8EDH5R+JfDMucI4ZpOzKN5SLD+j35QLLySgGXholoDxtqjHnmRjM1rpj5SIaXjHBZdJi7iax+1MRywHYBi5BfTtzYPgA+NH8mt+XkcdkEuFPcYxGh6fMQDQWb2g9Euze/6qi0W9mbYXvvb1/Q3bhVCXg1UgKD42BndWEpbMYFfzJLZWPtOD/lg81pwUmf/4BpPFv52sy87lQVfkkvcUliLObR4S0TI5hPLYxBS+RXmHYKgvPMB2QJ9wum62lJ+YM1MxPygS0OWbIB6UQGXmvxRXA4osXkxJ8ksPPCupOgFNiiYro91cQh3mmi6iHc9kMz5R1aNGGOfDdlnLx9DRSIiNcacjXEVNFaoM0x441f4GeU2ZR8/7uq2dfk7QjPlh1eJGxG9pzRiQ8JCDF1car0eYwXdMTsQoQqpIi5imqcELdcqGGp9FhzFElgUwpf15JYlkouVhS6hq4LDjhLNk7dDT8fmCEnHOPcbxaTgN7J3idnJK6zeIniuuk= root@iZj6ce071s2zz3reioxn93Z", "key_options": null, "keyfile": "/home/kevinz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kevinz", "validate_certs": true} 2021-08-21T14:32:36.574 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1YAaxAvlRgu72huAggHpoDKSgtF/b00ACfUX4Wk5NyLFbYI3PAQapHNcNL4mIsH8pc6YkdjcJ5N/F12aEupJGQ0Q06OS324xapnaBoUy1hQl9IvhtPqTLEwR0X5xLIAhNBLIEJDNVg13MKwr0u5YVUXdrQRu7Kj9RXAPBahNZFo/Ed+cnR3LICQWliHeHFrk/DU2DUHol3a5wAPxrowAG0AxAecP6/7rASMmitvLQwdQcfIJAhWYgfyXzudg+gOmAEgv3O3U4QOikgmkm/EabUXLJnlTj6yuhtuuctdJr73UT3wLGAGmphLKCkug27kb1nTNoDa+PW/+grP63zhU1 chunmei@otccldstore05", "key_options": null, "keyfile": "/home/liu-chunmei/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "liu-chunmei", "validate_certs": true} 2021-08-21T14:32:36.778 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLbPdDb+XHOD2fo8g3RacATj0cG07uELwUvqQP35y+JmhNRSb/ttaysFLA5ss0y77TiXx22saP3ECEm5SnuPg5IERkPQmXiw84ATTNfLnzx2G5H223SLaTQiGfSgniBCJHVPCJfNahJiurlZoOAlwDz6SKXTUiivVrFJ22iTBDCQUO6HFQKjE7aM3RAo5mZXKVlTXzM6WOd8vlzk3HMzV0/91JR6H86Lt9MEzXT4dOzqmtxd8gS6U+vA+50zOIspdK+orTRJ0PO2Fn8c1RSYM8g7uqQtpDcOEHderW8TRsqBbDxpyslK6hRX5urUNBX506XE6JNoViLNvCofgseCGj yingxin@yingxin-server", "key_options": null, "keyfile": "/home/cyx1231st/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cyx1231st", "validate_certs": true} 2021-08-21T14:32:36.987 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdqw+8an6CHqsagFqXfuzQi1zCFHCh0e9pDtYz+ylA7L+7vnJLjTR7km0bb4Z6VTUh7h0/nUfF6faPzD2KGmTKvS+mTpeOUaN/H8O8j8qqh90f8z4T8F8L3DwqQbPf/R0izXmTgz0FZObFHmj+G8a+D4ndxF80EPzKkML/G0Hg2GEVdYzazSY4imSY2f56ul6qByBknhFshaTo2DUDcdbDDokF6lO6WgXhY0MJrT24CUSMyZmyi+3hF7vowW2dfbJ+SuDft4XmPDnIxRhDWWvi12jZb8Le7SWYzUlu7fAGbWe63VJqNh992myI61aZshApDD24+1xqIoo+Z/XkQD3l sunnagar@localhost.localdomain", "key_options": null, "keyfile": "/home/sunnagar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunnagar", "validate_certs": true} 2021-08-21T14:32:37.196 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDTlsh9TFwLCeeD2RR+56VhRZ0hp9CHj72U3Z1gbMpxpcg6PYEDzb+0qPVQp1/cIhcwwy0sm13yFl/A5i0iCRpyqB2MbyaXjj0HgkNinxLBFkPSiW2C83lfAyAhG/gLk+FcmWE4+xVrE29aoSM0JtcNtp+q83UGdnFcvhFe0pgJbt5vITo3ZMJND/UZrzeewyK8Vy4Rvpfy8aHYi2Eq+7WQrRHwUwxn0CwsxldnUfmZtu58d+Rtt78LRF0mvcScNlC7q5D+8htmYwkWkK639w5IB3xPv+Z5cV49Psi24Q+9BIr9Zo9/OnnCTauAi5ivDlh+kk/iBksAE8rQpMhvfuSFnNoq2O87aPQvNQHdRCcOa7FVIDcuocu6AB8UP4cGcJreT4+BW5Zbk6vQeJZhmj9yExpRj9MuONy5793nKuAMPn9lOjuXRfK7A5Tb3OOhpEJK1/MjtdZz4mNmFFfFIZ24hgTZO0pOiMwDf/tQ6BG7H8NS9g3QmXPyZajUocw6Szs= lflores@redhat.com", "key_options": null, "keyfile": "/home/lflores/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lflores", "validate_certs": true} 2021-08-21T14:32:37.398 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGq2g8OFN5msjIn5CNB2BRss9eIyJRk3r+7RQcnBOQid jsawaya@redhat.com", "key_options": null, "keyfile": "/home/jsawaya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsawaya", "validate_certs": true} 2021-08-21T14:32:37.609 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNIdsy+yA0QjwkvjzTiH4lKB+txlh4oGkgC0r1l9swmFvD+MuzNNkD+oB/YscT51tJXsP5/wZDZMuk60xds4Os2ejoIBvzdGBgZ1G2B33etHPmzgTFE7ZegBkMlvWYmrJ9YS+iaMvXn0MMtqfE4zq0kemVAA4NBfFZMhHPJMKCg7Gj31MR0S0b7dcPx7Jnpg4A45oHr0qnufeqZYQaHGgo+LO2xdoV4V2EswWr7YG+b1wI6DzFZBbNsGyQd+cE1BzQ8NmETPNNpDPk5dCyBS7U+4ZTf/WbWk3+Cn+Ky47iWfxcscupQRkCjmTYI/uCMhnrpGU0zEHkbFwLTS6wqDtDmVHm++xof3xSsGJ2cE/vZySZGHqdxqVXfT8gt17h35rME8eX3TU392rjfK6kjEUD3UgwJJp4NFzQIEXi2HTmDYx/Vi9nuId3V1NrDz98QLPUd1ijiy1DuzI0BJPE/DTDmsb7hNXNIcAV8VRc1243W3eaee/Pdg/wmwfkrgjTU3s= shuaiwang@Shuais-MacBook-Pro.local", "key_options": null, "keyfile": "/home/markw8973240/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "markw8973240", "validate_certs": true} 2021-08-21T14:32:37.813 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+ZvZuogU+WfyGj4BR2xYllNoQqAoxQ6AM0IYddBV2pL1xgTM/SopuAxDRP0q2stpozMtjVjlvuiIgr/caqyTuOgYtYCvB09ppefqdnCDrJJBCOKpjUEnCpAKcBMMuz9V32Zqt5FgpIi0Mu7Gp/QCvB97imxEkoUunpdX/8Ri/xz0nJO+QqoJX5wOn91jG49XnkMCLzx+MuafhT90WFO9CV1sIVREZS+X2C7ACQuhx9Qwk7R4BXt2f3M4dA0K2nLwaAnVW50M1y6Gs50Wl+ojDR1itySOxv8YpwyRlRTOzQTyVMnG0Qsaxxwp+hy4b/vQW3NIUvHYdTyvqALTQdGHT lricomor@lricomor-mac", "key_options": null, "keyfile": "/home/luisrico/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "luisrico", "validate_certs": true} 2021-08-21T14:32:38.033 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKjxTgyhL1yFriyNvvuhUc6SE8+6f6lT6FTF8cZL9uiz matan.brz@gmail.com", "key_options": null, "keyfile": "/home/matan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "matan", "validate_certs": true} 2021-08-21T14:32:38.248 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkFTEUxQkQ1cplOLOf86Teu4qTYGkMqWtjIwzsU/AcimjFolSI5fJ3B72PH7bhQKkSnOlTjiOos2LrD5i/XgxkJD6icVWwHO11lczsq/8K5x0ZMJbJaiT3QphBB4yItoge2Y4/7g4o/78Ne+U7eZ1L5fgeO1fExtIaEXK1lS8pVlpZQXyproTIzp1WLHoaXb38Mxc5jAtuvEKfEycdX3c8k/9ti2qBCZnmjPH05WOxkqCRyNBT+bSJZyFMYt9RDUuycn3BYVSXTAzPW0qwgvKY5vij+qiFoxij7nKNMH7w6C0arolCcJxhiMyEF3UbTuOxgVONWyqFk/vScGGSYj9ytQYjv0L1cFyOVhlpbu+tnptulfQ3faMpjROE/eYxTIoY7OPDgWbFqEUJ4eEhKwAC7aYIYUhvoI/8k+4u64nsyEJQdHXmB+OY3WE5TegYUCuqSsdPY74eO1o8dDgxF363eggDSBmYqSSs+Tpu9TNhJG7iJFX1tiAPwmKvjVrJOac= ozeneva@omri", "key_options": null, "keyfile": "/home/ozeneva/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ozeneva", "validate_certs": true} 2021-08-21T14:32:38.452 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDepyzA+MeQT5oRM52SxmUUKVAzkcrH/NqFxoCpBAjUSioTMaB1h0PhHE00nvHnKRvqWdPl8LCBWS3Ncjh30r9lwHaP+oPy4iLQEc7D2Mwn7WyPh5cq9N3UF25C0AlMG744k1tBNb97qy73OJI+oc5pAj4vUpCC1Ut9pMP5E5pQCx/kwgSOpC7mk0dYr/H5FxjPb0ca55eisFTLXraHPSChdrHjlYC5XpOZ9csilTwFdF0fJdu0nMDwIxGvwTOaUcIcfNLkpGB2Sj/pAM4L9pVsIbeE3C8d978MZ8Ff+ksQ0T8eNzew2ZjSkSuV+oR1sBCJYZ7mbDQuYLKAs5DanO1VFJvgjBJm1zP2ONu+g0fn/j04qIosuudziwoG/dOTkFo+6dBvF5hQi7TbZOIUVFr5dBlGS+O4eoK0gi/wpzun227E4dadY9uB1wiZ/PTD/5og+gxWG32RkHqqX3ALlK4m8JKzrq214nZ7Dt9CNewiUkR15uRkDasGIdiu6qb4Ez8= ArunaSudhan@sudhan-linux", "key_options": null, "keyfile": "/home/sudhanaruna/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sudhanaruna", "validate_certs": true} 2021-08-21T14:32:38.672 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuHIpWAz6Sk3ATLvTG4KS24JUzgqQBEm4EesPgf1cEG+xK4gISUixaVC9OSNS/KR+8/yoWAhmw992p8cp7mWXxqrRC9kIrthLWvD6Jjz3HTXgarO2oNibeVu27kK/Fm2vvUNACARBkoC5YXd6IuoZOEg1bJu06DLw5Y7pGMqZRVDOrE8WUDwr27roZSgU2/N+18Y1LYn4gwSvWpuC/IotLyGSpW7/dgrZAIreRoYboMdTSlIjjuzeWvYx/AnZmkeEo0IhfyuBRl7ZQwxSFYHn9Z1+iQkt+O9hEZYSZuYRuyFPRqHNMKCEj1iYVRMcwbf/x2/dhjfmbWmjOMJoNNwm+DpixHHbId6Idx82GlAtCgQSTe9aG783oCeZYuVUr1plIRjeEsdjY9UBRYKNknEAChKjMSbxysvO3Dy4yrU2r8qo1iBS1fi4HUg12asCLrJX7vSCvOhOn7ltqDTmt5UiLSn+eCzXpOTBDWz7McB28FoYyR+Rwz5GOH3ITFhPyhXc= wzl@manjaro-wzl", "key_options": null, "keyfile": "/home/zl31wang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zl31wang", "validate_certs": true} 2021-08-21T14:32:38.873 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIi8JMOVD91QR7EXBkhxeAsB2UO2gyWqYPiP6HjJO/TzzMKf7QtO928XzdORwQ1/jm2hDF1kaNPaU/XIHD0hHSvfICS4ELQcBPa5uPya6d5Kn5fVlpGqBcXBxKGbHB01PGjn0214eupt2LebWxkm7TP/BGDHKXXaoR748AlbyyG2L3TaNoGkwQiEoZe3CWgH4qkrf2CEpyfOa7BM7U0h+lceJxIGe+Yo6Ll14zyngDBa059E+ndFTpkQ6vECII7FdB1+A4Orhu40XHZgyVnlRudD1l80pIVnZves8N2zHZhQo8hp0sB5oY8HENiB09j7Xa6YQ/c1c2wm9E2xJHSNo7muFiHz48vQEgvNmu0lZdbVwXIy3IgLEBHjs6Kyqjtg618V4UaoP/LBOqtWtLd6TvGJ3qBlNyvhO5gZ2DGeLMN2WWoHxfl3VM+QGCGI2dnzUyHhfG1VBDYj8fzWXOuUVpVCMUw8fEg8GWmYLDwm3SbdMACWU87J0kAmaDPKcwwL6AwtWJZr/p89sSypl6y50JlOoDrbDbxrIbHQgbf1tcEgPOuLUN+O06qpi+OxKMtFdbvz+NQ4V4CpXpvMMQZj5AbuHp0EJ5Vme/Kcmpl/xoYGyFDzOEGR8lw6LMz+CpQ4GtnK5XXe1Xn3TJh6Tn2tVyS20p7artemGQQquCN8iQOw== mayag@gett.com", "key_options": null, "keyfile": "/home/mayagilad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mayagilad", "validate_certs": true} 2021-08-21T14:32:39.095 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFG1CoPZEh4KBANdJZIzLsutHsF6tAYZTdDdEW+5E+8YyNHOy42D8iBLg+eCAbIl4SFmJTsPNtyO4L8/Ut79tkG3usA9nuGoAwGYuctZUTgOiTiV61940Y+NTvwan9vFNoknVi/3qIGNSXdm90I1aWRTKLCXRbGA1qvu+qyJv93O8LFaPni2TJmtAebY/xf0fHhW2fURUoeoCsoobrTkPA8SojT0A3SM+31iOD7sxbmsqLBR790wz/FHh152PiqpupIijS9TaZgGuVpejM3rgAC7H2kVmKJlrmDlmfcbKLS6BBi+uUDi6ta8whnDRq03nP/4LJE7ZDIxX3I4mHztZeJggURlsfpJdo99+VHkeWlGDBIP41QgD4UX1neuMcIO3nuEnZmLeJyvo9JOR/I7AudbQCDPjSMNbrF+el70mWLhvPe33NA3x6eOBRPc3r+qrSwMhXEPatZlbywJRY+O6yC2TYqdlUKsUEVA/D0Iw7tO+pf3yH3Y/iUDJJLvViRZs= Arjun@localhost.localdomain", "key_options": null, "keyfile": "/home/arjsharm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "arjsharm", "validate_certs": true} 2021-08-21T14:32:39.316 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCXZDAUNEK4+/6Y+ywEtFnabnfvZmzzHhO6y4PbRHgx8SoQ8+7au/5ElB0vjjiJJSChnDymdqP0UzwcShzhWA8is6K9c6tXUHg6NcgV5tN2xYYmGIIWwm66Z1kYH6u1x2Jkv9LO04rXgebkzXPPYK401agCWjkjUv5acZE12+M/B6d4h2lpFjzBqZ/mmRNp0tHdCapdG/nS1i6GRi3Yi3fQ89CbiZArG9268DSEnLe+npeyBMVdDkWpVIytQ6pU1+g0prrePIVaMvrK3yEooduD4HeLr4liqkINj6rCs90xedPcFi0++UaLFRcgWh3OOnrI952Oe19dx5hlbmcnMPGjaXnkng2keWFyZMOQRRti5l8C/TXPUTaYGWEoqFEEabIKnCura0NWuZHpeF9xEK0tnvNVeauuUfkPxWQ7PH8QQ/9XjOkKbmy0+2q9tug8RC5i5NwcRE4W3ijM7pLwhWMdxkmIOe3w1taPq8HU95hMQchl2749oj8DRyVePVK9aH0= melissali@melissali-ubuntu", "key_options": null, "keyfile": "/home/melissali/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "melissali", "validate_certs": true} 2021-08-21T14:32:39.525 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYdmNB3np8OLc49B4cAh59CdNYIb4fAq2XBQ1sbOZMMZmhpFobyy+i6s4lBnovMNWAB6HmpsECHrKRU4IFl9lQ7vV+++J08SLoAkl6G0Gk9tmbgEs+/ovCZg+qUfZA1ZccSAPHmtqfZOiJcnlIlhQq2BoFnLUwmpeIQ4H5D+Y2q7DMdyGkdCenH80aD5KuiwLIyiBL9q7RYz0U5cdCO3wDFU4+5XuZiT8VYDUgp3mWPpAiPdGsJmCn41tpt3PViPDQlKh4vLW0ieQUJI/v2f6oHbM9YkxSoVlwLcrjTv+Nl+dVymqVTF4UaT1kz+ZnoH6khKZxSHvKm9YHc4LNZKYoVdyCxdZvBfxbEbXpT702vLqDNGEfZSlzJs5kVqP30m8fI/yzTQeD7tgjzzrKWdLZacyo+9cWeH6nFfZAQuZ9ug1lv3UAs5x6MsidEH5u6Iymi8BE+mUsOXnSpSfUDDo5n6+CMVe93brdW21iSX9Hbuz5RKB6xile1ROHpvpi1u7VQ9Hp3FIZFFiix9S5jniyB2OqjGKTO6RULfPsJo8sXCLq9YKq9zrkpMv5wURMQKnjtS5OHUWh0cde3U/VikPIfLkzEsfLV8ICAksyuWHQJGVUMxhPmSTKtRsPScbklekT+n65rexDt4Qr73YOy9BUIPV4Mo+kkl4ugQd6VCKJ6w== ibalutoiu@cloudbasesolutions.com", "key_options": null, "keyfile": "/home/ionutbalutoiu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ionutbalutoiu", "validate_certs": true} 2021-08-21T14:32:39.745 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDuyLB5rnyjlglcHVCoCSG7pDVJQcTYq/I2yGr3ohK9p6boWxk+zW/a2Q7+qNdKk2DtUC4WvvZ76BU56FC6/LPIUKSJLbN/Xv9/dnZcqKf+ZVlCQgMrE8yGoWt8WLtejJ1iBj0EdoDfYwDiL09ygUim1L1hNxFDOVq9RA2Lo6Tqk5lcAgjoCnU4NZY2YzTXOUSZTzsv/1bd6WVudKHJuWF6ZPb+mbbMoWqwnUZCHwjW8PgTXaPhk1929P0ftWT4ncfKnCc46LXf8OkvzQgCrLyh8964tGKPWppp3wNg2AJ/zFvRY8kkJZSxOawp9Gu8+kCNwWUmLvE9bqYJKT7EMDrU+8TMUEWLFl5fGoxoLWD113PPBcFrpUhvRAJa625eec8a/inv4cqPf6dY6qyXAReXURJm7LnHT4fDq+F6op8+tmZT6Tzkw4+YMUd9h0XDJ4pnPQpk5tjWER/HuWm73+ialXo+ERQWnnVCB1IXq1FEOeF8YAslKC7aSYIZjG9uUkM= peristocles@localhost.localdomain", "key_options": null, "keyfile": "/home/pdiazbou/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdiazbou", "validate_certs": true} 2021-08-21T14:32:39.954 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2iBYL614KRYvFhow408yzMljYRdfyMmbnkUVN8waja4RCsmxTpvIU1aBQ02bMTsY/0n+WMWAbI1D6C1HbuG+JbFYBLSgPCTSG8FIpukFU8y1by8piPu2/ZBD1XdFcPO6u1Nr/tZC08EXhfqvnDAVJUyg6rLLs5+TFKAiPSuv5GyrMRWvMI8eoPk+4u2r/ThxXZTDaIfJhPXlULMLL5ZxR4KZU+WtWbCGSFDGjSugCqdTzpZyX5jUq9C9JB83RxojCtI5n9rCoXLcGOvW9HBnVMxhS56zE8BwxLxLhGy6S1lOoyBPzu7w6XIc160JjCK5iAmlXRqG93nWEVrhZkxCz wangxu298@foxmail.com", "key_options": null, "keyfile": "/home/xuwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuwang", "validate_certs": true} 2021-08-21T14:32:40.048 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:40 +0000 (0:00:32.820) 0:01:18.314 ******* 2021-08-21T14:32:40.150 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-08-21T14:32:40.152 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.162 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.163 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.175 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.177 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.197 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.198 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.408 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-08-21T14:32:40.419 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.432 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.645 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-08-21T14:32:40.847 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-08-21T14:32:40.857 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.879 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.881 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.891 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.902 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.913 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.914 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.924 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.926 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.936 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.947 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.948 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\nyehuda@crestview yUJ7nsAXRv78MgIyxJ93Yw 5b872feb20cbd895a5d8c87956f9cbd491471654b1a29aa8c13826cd95a44c40\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.958 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.960 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.970 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.971 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.981 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.993 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:40.994 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.004 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.006 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.016 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.018 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.028 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.041 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.052 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.063 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.065 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.075 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.077 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.087 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6\nprsrivas@t480 jciDaqMz2kZbSRymwpJiqA 68baba83dd966fe9536bc9a822428710b10a28781f695bc3fb750482a94b0e8c\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.089 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.099 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.109 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.111 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.121 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.132 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.133 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.143 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.145 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.155 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.167 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.180 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.181 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.195 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.206 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.209 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.217 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.229 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.230 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.231 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.247 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.254 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.256 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.266 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.277 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.279 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.288 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.299 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.302 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.312 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.324 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.325 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.325 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.337 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.349 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.351 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.361 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.361 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.384 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.385 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.385 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost w/vFHwM/x4hUGlKJCteKiA 97933f044fdd8b3cf57ad744ce032728b06f354b0d719790def82b2f5b9a6133"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.397 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.398 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.409 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.421 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.432 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.443 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.445 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.455 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.466 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.477 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.489 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 k0lrZNIKWMH/VrZupJ9diA 03bfe34ee6e9a541e10426a3cc1e1046655420d4d0a79ce5667bec1b469a0100"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.500 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.511 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.522 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.524 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.534 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.551 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.557 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.568 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.570 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.580 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.591 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.616 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.625 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.636 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.638 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.648 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.662 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.664 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.676 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.678 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.688 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.691 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.700 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.711 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.713 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad 6vBnL22Dfp8d9grCNtSoHw c8270453301dbec76f8070809078087ff892c74360b80a82002d58c50af19ec4"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.723 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.734 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.746 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'dachary', 'ovpn': 'loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dachary", "ovpn": "loic@tulipe 1NGxatlPDW6m9FLwzG2Gng cc19cfe2d2101909e9e0ae6b207d7e8dccafd8ec1f4594bdd006dda6b67a80b5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.749 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'amctagga', 'ovpn': 'amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amctagga", "ovpn": "amctagga@redhat20 DsFh00Ttjtomgs5nlTX3SA 713280e601e14a8fecfe190490e362f5f97bba3517b2bcfd4abd5230a45fbaa1"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.757 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'kevinz', 'ovpn': 'kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kevinz", "ovpn": "kevinz@linaroceph a2GchLhkaayg78xuUnI0sA b545e84634e614099b00d50e593933eabf0793035eaa439b6d82ec669f69de7f"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.768 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'liu-chunmei', 'ovpn': 'liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "liu-chunmei", "ovpn": "liu-chunmei@otccldstore gKMLlCSH99LCDse00ja0Kg 2c140368bca0e48102330631f91e5299298461a24cab483b9765293eafb2050c"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.780 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'cyx1231st', 'ovpn': 'cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cyx1231st", "ovpn": "cyx1231st@yingxin-server C3qUn5JXYFehgyfol2lnhw c5f306797eac838797ea99b07bd0f5e8e185052887b82ba5c343a42eb271747a"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sunnagar', 'ovpn': 'sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunnagar", "ovpn": "sunil@sepia-lab j6h8NzyTGEE6WHYT3uiOXA eda7763cdd544d8e7198cefee77f38641d44c8a6aa4d37b8b1429620641809ae"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.801 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'lflores', 'ovpn': 'lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lflores", "ovpn": "lflores@fedora CqF8+5Scni+GHCfHkm3AAA 0771c83aa58141af4704f4b01bba4490638ee2033fdabe5a2a4b8e06768b29c2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.802 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'jsawaya', 'ovpn': 'jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsawaya", "ovpn": "jsawaya@thinkpad jEJaJNljo4e2sHstFCNljQ d0042e4c40344f7fa350d3e0b748686aea00c4757ffbe32a41f3b119463d9f46"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'markw8973240', 'ovpn': 'shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "markw8973240", "ovpn": "shuai@mac d7Tqhwnt5D7+jeeqKQ375w e6ac2fb0750350a007d00bba1354989402199dc012150a84805be57a92e6f06d"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.817 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'luisrico', 'ovpn': 'lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "luisrico", "ovpn": "lricomor@lricomor-mac DI4SNq2EznedeK8Cztt2ug e05bc16840e8c3c45f641852101ad921abc248013ec89a599c27c275767b5347"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'matan', 'ovpn': 'matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "matan", "ovpn": "matan@main-win vEAxj3ZM9DjxBi/jfjc+jQ 9c30b8fcef9bdc804090947a92e1e60cc21d3f280f261bc3f4cecac64541dc6e"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.825 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ozeneva', 'ovpn': 'ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ozeneva", "ovpn": "ozeneva@omri qB3om9JUxgVneOibFeRq6Q 92277afcd0bce888f60dceab4b2e8695ebbf7241dba09e3350a12c4fffd6ebd5"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.826 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'sudhanaruna', 'ovpn': 'ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sudhanaruna", "ovpn": "ArunaSudhan@sudhan-linux t/fhf5WR4vQxW3F/Lx7Atw 09248b17c9abfe802ca4be1b93ad318138f54fb60b073ff857c2effadb27a997"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.836 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'zl31wang', 'ovpn': 'zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zl31wang", "ovpn": "zl31wang@sjtu Et6K7OVsdRTnvOY5iuvUxA a7f66c06b405df269ac8f28a4b2c9f00d087091082c53ed5e03517e3811d9fc2"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.847 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'mayagilad', 'ovpn': 'mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mayagilad", "ovpn": "mayagilad@mayagilad-ThinkPad-T480 AGw0uOFkRU6SX8Kp3nyGuA d29a792aad9cc6038dfc2e3471f6556b50ef73562b1bc6231c39d8daf20a19d0"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.849 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'arjsharm', 'ovpn': 'arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "arjsharm", "ovpn": "arjun@thinkpad_local 06szrDO7vKFS7BSicSbDsw e1284059b3e0604191a0f35e2841f22835b0e75249e637d8297edbee458ccf39"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.861 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'melissali', 'ovpn': 'melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "melissali", "ovpn": "melissali@melissali-ubuntu pooAecOHrOtZ3gXewVMxnw 36860a4078c773f47e1a58d2f120ec79e468142120ff6a816cb6ffbb9fd9ef07"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.863 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'ionutbalutoiu', 'ovpn': 'ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ionutbalutoiu", "ovpn": "ionutbalutoiu@macbook ruUdfqkSfEhKctGGLeQvkw 0a6cab2332041d56ca8b1216242195af2ad8d0ec6360fd6c1dd080f5f43240e1"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.875 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'pdiazbou', 'ovpn': 'pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdiazbou", "ovpn": "pdiazbou@thinkpad BNtoPcISV462wNeprZDOcg f6faca031e6549014132b4c66e53721c9d934a09d0d9d0f36e2dd31f90be1ce4"}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'name': 'xuwang', 'ovpn': 'xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuwang", "ovpn": "xuwang@laptop 2U0uRjvVh8eE4iSkFzK2Kg 2a9db1c375a032cc0a892ca2d0721f9d7303e2db5a7eab647870d7c2022ee2e8"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:41.940 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:41 +0000 (0:00:01.892) 0:01:20.206 ******* 2021-08-21T14:32:42.020 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-08-21T14:32:42.021 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:42.046 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:42 +0000 (0:00:00.105) 0:01:20.312 ******* 2021-08-21T14:32:42.337 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi077.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-08-21T14:32:42.530 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-08-21T14:32:42.720 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-08-21T14:32:42.912 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-08-21T14:32:43.113 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-08-21T14:32:43.305 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-08-21T14:32:43.494 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-08-21T14:32:43.685 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-08-21T14:32:43.874 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-08-21T14:32:44.064 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-08-21T14:32:44.259 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-08-21T14:32:44.458 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-08-21T14:32:44.658 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-08-21T14:32:44.869 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-08-21T14:32:45.056 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-08-21T14:32:45.250 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-08-21T14:32:45.456 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-08-21T14:32:45.636 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-08-21T14:32:45.841 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-08-21T14:32:46.041 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-08-21T14:32:46.250 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-08-21T14:32:46.441 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-08-21T14:32:46.645 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-08-21T14:32:46.835 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-08-21T14:32:47.032 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-08-21T14:32:47.230 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-08-21T14:32:47.425 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-08-21T14:32:47.624 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-08-21T14:32:47.814 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-08-21T14:32:48.021 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-08-21T14:32:48.213 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-08-21T14:32:48.409 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-08-21T14:32:48.610 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-08-21T14:32:48.802 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-08-21T14:32:48.984 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-08-21T14:32:49.185 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-08-21T14:32:49.374 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-08-21T14:32:49.562 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-08-21T14:32:49.763 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jdillaman) => {"ansible_loop_var": "item", "changed": false, "item": "jdillaman", "name": "jdillaman", "state": "absent"} 2021-08-21T14:32:49.966 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-08-21T14:32:50.164 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-08-21T14:32:50.355 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-08-21T14:32:50.547 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-08-21T14:32:50.745 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-08-21T14:32:50.954 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-08-21T14:32:51.150 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-08-21T14:32:51.334 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-08-21T14:32:51.525 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-08-21T14:32:51.728 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-08-21T14:32:51.817 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:51 +0000 (0:00:09.768) 0:01:30.081 ******* 2021-08-21T14:32:52.419 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-08-21T14:32:52.421 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "checksum": "39e5632aabd93a7648a3940c07007a14afe2d320", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "1f362c6963bf1f0d83c5c22328e42c02", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1629556371.9248514-1318-165209206943454/source", "state": "file", "uid": 0} 2021-08-21T14:32:52.443 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:52 +0000 (0:00:00.627) 0:01:30.709 ******* 2021-08-21T14:32:52.512 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-08-21T14:32:52.513 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => { "msg": "Host smithi077.front.sepia.ceph.com is running CentOS 8.3 (n/a)" } 2021-08-21T14:32:52.535 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:52 +0000 (0:00:00.093) 0:01:30.802 ******* 2021-08-21T14:32:52.621 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-08-21T14:32:52.649 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:52 +0000 (0:00:00.113) 0:01:30.915 ******* 2021-08-21T14:32:52.720 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-08-21T14:32:52.722 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:52.742 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:52 +0000 (0:00:00.092) 0:01:31.008 ******* 2021-08-21T14:32:53.060 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.083114", "end": "2021-08-21 14:32:53.031496", "rc": 0, "start": "2021-08-21 14:32:52.948382", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-08-21T14:32:53.080 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.338) 0:01:31.347 ******* 2021-08-21T14:32:53.151 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-08-21T14:32:53.153 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.177 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.096) 0:01:31.444 ******* 2021-08-21T14:32:53.252 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-08-21T14:32:53.261 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.285 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.107) 0:01:31.551 ******* 2021-08-21T14:32:53.370 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.402 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.117) 0:01:31.669 ******* 2021-08-21T14:32:53.474 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.501 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.095) 0:01:31.764 ******* 2021-08-21T14:32:53.603 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-08-21T14:32:53.604 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.628 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.130) 0:01:31.894 ******* 2021-08-21T14:32:53.714 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.747 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.116) 0:01:32.011 ******* 2021-08-21T14:32:53.853 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-08-21T14:32:53.854 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:32:53.878 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.133) 0:01:32.145 ******* 2021-08-21T14:32:53.948 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:53.981 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:53 +0000 (0:00:00.093) 0:01:32.238 ******* 2021-08-21T14:32:54.048 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.071 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.099) 0:01:32.338 ******* 2021-08-21T14:32:54.142 INFO:teuthology.task.ansible.out: TASK [common : Set the releasever] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.165 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.093) 0:01:32.431 ******* 2021-08-21T14:32:54.236 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:32:54.267 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.097) 0:01:32.529 ******* 2021-08-21T14:32:54.332 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-08-21T14:32:54.334 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.357 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.092) 0:01:32.622 ******* 2021-08-21T14:32:54.430 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:32:54.453 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.097) 0:01:32.719 ******* 2021-08-21T14:32:54.523 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.550 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.092) 0:01:32.812 ******* 2021-08-21T14:32:54.623 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-08-21T14:32:54.625 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.645 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.099) 0:01:32.912 ******* 2021-08-21T14:32:54.727 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.749 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.104) 0:01:33.016 ******* 2021-08-21T14:32:54.830 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-08-21T14:32:54.832 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:54.856 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.107) 0:01:33.123 ******* 2021-08-21T14:32:54.934 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:32:54.957 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:54 +0000 (0:00:00.098) 0:01:33.222 ******* 2021-08-21T14:32:55.050 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.076 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.121) 0:01:33.343 ******* 2021-08-21T14:32:55.161 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-08-21T14:32:55.162 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.184 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.108) 0:01:33.451 ******* 2021-08-21T14:32:55.267 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.295 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.106) 0:01:33.558 ******* 2021-08-21T14:32:55.365 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-08-21T14:32:55.370 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.389 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.097) 0:01:33.655 ******* 2021-08-21T14:32:55.458 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.486 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.097) 0:01:33.752 ******* 2021-08-21T14:32:55.557 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-08-21T14:32:55.564 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.585 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.098) 0:01:33.851 ******* 2021-08-21T14:32:55.672 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-08-21T14:32:55.696 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.110) 0:01:33.962 ******* 2021-08-21T14:32:55.779 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-08-21T14:32:55.781 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.805 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.110) 0:01:34.072 ******* 2021-08-21T14:32:55.890 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:55.910 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:55 +0000 (0:00:00.105) 0:01:34.177 ******* 2021-08-21T14:32:56.192 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-08-21T14:32:56.194 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:32:56.218 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:56 +0000 (0:00:00.304) 0:01:34.481 ******* 2021-08-21T14:32:57.094 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-08-21T14:32:57.559 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-08-21T14:32:57.590 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:57 +0000 (0:00:01.369) 0:01:35.851 ******* 2021-08-21T14:32:58.125 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* ok: [smithi077.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-08-21T14:32:58.564 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-08-21T14:32:58.588 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:58 +0000 (0:00:01.003) 0:01:36.855 ******* 2021-08-21T14:32:58.660 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:58.683 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:58 +0000 (0:00:00.094) 0:01:36.949 ******* 2021-08-21T14:32:58.765 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:58.790 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:58 +0000 (0:00:00.106) 0:01:37.056 ******* 2021-08-21T14:32:58.870 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:58.894 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:58 +0000 (0:00:00.102) 0:01:37.158 ******* 2021-08-21T14:32:58.962 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-08-21T14:32:58.964 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:58.990 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:58 +0000 (0:00:00.094) 0:01:37.253 ******* 2021-08-21T14:32:59.078 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi077.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.090 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.116 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:59 +0000 (0:00:00.126) 0:01:37.380 ******* 2021-08-21T14:32:59.184 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.208 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:59 +0000 (0:00:00.094) 0:01:37.474 ******* 2021-08-21T14:32:59.277 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.301 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:59 +0000 (0:00:00.093) 0:01:37.568 ******* 2021-08-21T14:32:59.370 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.392 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:59 +0000 (0:00:00.090) 0:01:37.659 ******* 2021-08-21T14:32:59.470 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.488 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:59 +0000 (0:00:00.096) 0:01:37.755 ******* 2021-08-21T14:32:59.549 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-08-21T14:32:59.551 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:32:59.572 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:32:59 +0000 (0:00:00.083) 0:01:37.839 ******* 2021-08-21T14:33:00.526 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.759535", "end": "2021-08-21 14:33:00.500353", "rc": 0, "start": "2021-08-21 14:32:59.740818", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:33:00.556 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:00 +0000 (0:00:00.982) 0:01:38.822 ******* 2021-08-21T14:33:04.479 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:33:04.509 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:04 +0000 (0:00:03.953) 0:01:42.776 ******* 2021-08-21T14:33:04.581 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:04.606 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:04 +0000 (0:00:00.096) 0:01:42.872 ******* 2021-08-21T14:33:04.717 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-08-21T14:33:04.719 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:04.746 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:04 +0000 (0:00:00.141) 0:01:43.013 ******* 2021-08-21T14:33:04.848 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-08-21T14:33:04.850 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:04.881 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:04 +0000 (0:00:00.127) 0:01:43.140 ******* 2021-08-21T14:33:04.948 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-08-21T14:33:04.951 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:04.971 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:04 +0000 (0:00:00.097) 0:01:43.237 ******* 2021-08-21T14:33:05.489 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-08-21T14:33:05.513 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:05 +0000 (0:00:00.542) 0:01:43.780 ******* 2021-08-21T14:33:07.095 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-08-21T14:33:07.097 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:33:07.118 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:07 +0000 (0:00:01.604) 0:01:45.384 ******* 2021-08-21T14:33:07.666 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-08-21T14:33:07.668 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-08-21T14:33:08.114 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-08-21T14:33:08.546 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-08-21T14:33:08.568 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:08 +0000 (0:00:01.450) 0:01:46.835 ******* 2021-08-21T14:33:08.764 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-08-21T14:33:08.767 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:33:08.786 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:08 +0000 (0:00:00.218) 0:01:47.053 ******* 2021-08-21T14:33:09.294 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-08-21T14:33:09.296 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "state": "file", "uid": 0} 2021-08-21T14:33:09.715 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-08-21T14:33:10.142 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "2fbc67b36d59f66fc2621f3ca2dcf60dea2167a5", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3612, "state": "file", "uid": 0} 2021-08-21T14:33:10.163 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:10 +0000 (0:00:01.376) 0:01:48.429 ******* 2021-08-21T14:33:10.234 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:10.262 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:10 +0000 (0:00:00.096) 0:01:48.526 ******* 2021-08-21T14:33:10.340 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-08-21T14:33:10.370 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:10 +0000 (0:00:00.110) 0:01:48.636 ******* 2021-08-21T14:33:10.613 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* changed: [smithi077.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-08-21T14:33:10.645 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:10 +0000 (0:00:00.272) 0:01:48.909 ******* 2021-08-21T14:33:10.750 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-08-21T14:33:10.753 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:33:10.773 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:33:10.775 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:33:10.798 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-08-21T14:33:10.819 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:10 +0000 (0:00:00.176) 0:01:49.085 ******* 2021-08-21T14:33:12.430 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:33:12.450 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:12 +0000 (0:00:01.631) 0:01:50.717 ******* 2021-08-21T14:33:12.547 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-08-21T14:33:12.553 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:12.589 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:12 +0000 (0:00:00.139) 0:01:50.856 ******* 2021-08-21T14:33:13.179 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-08-21T14:33:13.224 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:13 +0000 (0:00:00.629) 0:01:51.485 ******* 2021-08-21T14:33:13.779 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-08-21T14:33:13.815 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:13 +0000 (0:00:00.596) 0:01:52.082 ******* 2021-08-21T14:33:13.946 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi077.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-08-21T14:33:13.951 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-08-21T14:33:13.979 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:13 +0000 (0:00:00.159) 0:01:52.242 ******* 2021-08-21T14:33:14.248 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:33:14.317 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:14 +0000 (0:00:00.310) 0:01:52.552 ******* 2021-08-21T14:33:14.616 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi077.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-08-21T14:33:14.661 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:14 +0000 (0:00:00.372) 0:01:52.925 ******* 2021-08-21T14:33:14.788 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:33:14.823 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:14 +0000 (0:00:00.149) 0:01:53.074 ******* 2021-08-21T14:33:15.417 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-08-21T14:33:15.441 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:15 +0000 (0:00:00.632) 0:01:53.707 ******* 2021-08-21T14:33:16.312 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-08-21T14:33:16.314 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/nrpe (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1240", "ExecMainStartTimestamp": "Sat 2021-08-21 14:30:49 UTC", "ExecMainStartTimestampMonotonic": "19663736", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Sat 2021-08-21 14:30:49 UTC] ; stop_time=[n/a] ; pid=1240 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "984", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "L 2021-08-21T14:33:16.315 INFO:teuthology.task.ansible.out:imitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "1240", "MemoryAccounting": "yes", "MemoryCurrent": "671744", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "PIDFile": "/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "on-abort", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStar 2021-08-21T14:33:16.317 INFO:teuthology.task.ansible.out:tUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Type": "simple", "UID": "988", "UMask": "0022", "WatchdogTimestamp": "Sat 2021-08-21 14:30:49 UTC", "WatchdogTimestampMonotonic": "19663764", "WatchdogUSec": "0"}} 2021-08-21T14:33:16.342 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:16 +0000 (0:00:00.902) 0:01:54.609 ******* 2021-08-21T14:33:16.589 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001737", "end": "2021-08-21 14:33:16.566654", "rc": 0, "start": "2021-08-21 14:33:16.564917", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-08-21T14:33:16.615 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:16 +0000 (0:00:00.269) 0:01:54.878 ******* 2021-08-21T14:33:18.220 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:33:18.240 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:18 +0000 (0:00:01.627) 0:01:56.506 ******* 2021-08-21T14:33:55.315 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-08-21T14:33:55.317 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-67.el8_4.1.noarch", "Installed: selinux-policy-3.14.3-67.el8_4.1.noarch", "Removed: selinux-policy-3.14.3-67.el8.noarch", "Removed: selinux-policy-targeted-3.14.3-67.el8.noarch"]} 2021-08-21T14:33:55.341 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:55 +0000 (0:00:37.100) 0:02:33.607 ******* 2021-08-21T14:33:56.080 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-08-21T14:33:56.082 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-08-21T14:33:56.103 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:33:56 +0000 (0:00:00.762) 0:02:34.370 ******* 2021-08-21T14:34:11.028 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-08-21T14:34:11.030 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:14.684079", "end": "2021-08-21 14:34:11.007447", "failed_when_result": false, "rc": 0, "start": "2021-08-21 14:33:56.323368", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-08-21T14:34:11.062 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:11 +0000 (0:00:14.952) 0:02:49.322 ******* 2021-08-21T14:34:11.573 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-08-21T14:34:11.595 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:11 +0000 (0:00:00.539) 0:02:49.862 ******* 2021-08-21T14:34:11.849 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-08-21T14:34:11.851 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.009460", "end": "2021-08-21 14:34:11.829057", "rc": 0, "start": "2021-08-21 14:34:11.819597", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:34:11.875 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:11 +0000 (0:00:00.279) 0:02:50.141 ******* 2021-08-21T14:34:12.124 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.016428", "end": "2021-08-21 14:34:12.099371", "rc": 0, "start": "2021-08-21 14:34:12.082943", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:34:12.150 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:12 +0000 (0:00:00.275) 0:02:50.416 ******* 2021-08-21T14:34:27.271 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:14.885723", "end": "2021-08-21 14:34:27.244745", "rc": 0, "start": "2021-08-21 14:34:12.359022", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:34:27.294 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:27 +0000 (0:00:15.144) 0:03:05.561 ******* 2021-08-21T14:34:27.535 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-08-21T14:34:27.537 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-08-21T14:34:27.559 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:27 +0000 (0:00:00.265) 0:03:05.826 ******* 2021-08-21T14:34:27.628 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "dash_stream": "", "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-08-21T14:34:27.653 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:27 +0000 (0:00:00.093) 0:03:05.920 ******* 2021-08-21T14:34:27.725 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-08-21T14:34:27.728 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-08-21T14:34:27.751 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:27 +0000 (0:00:00.098) 0:03:06.018 ******* 2021-08-21T14:34:27.837 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-08-21T14:34:27.839 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ceph/python3-asyncssh"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["redhat-lsb-core", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist", "CentOS-PowerTools-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}, "CentOS-PowerTools": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}}, "a 2021-08-21T14:34:27.840 INFO:teuthology.task.ansible.out:nsible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-08-21T14:34:27.861 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:27 +0000 (0:00:00.109) 0:03:06.127 ******* 2021-08-21T14:34:28.083 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-08-21T14:34:28.085 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-08-21T14:34:28.106 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:28 +0000 (0:00:00.245) 0:03:06.373 ******* 2021-08-21T14:34:28.297 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-08-21T14:34:28.320 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:28 +0000 (0:00:00.213) 0:03:06.587 ******* 2021-08-21T14:34:28.559 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "name": "ubuntu", "state": "present", "system": false} 2021-08-21T14:34:28.582 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:28 +0000 (0:00:00.261) 0:03:06.849 ******* 2021-08-21T14:34:28.653 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-08-21T14:34:28.655 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:28.676 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:28 +0000 (0:00:00.094) 0:03:06.943 ******* 2021-08-21T14:34:28.745 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-08-21T14:34:28.747 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:28.771 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:28 +0000 (0:00:00.094) 0:03:07.037 ******* 2021-08-21T14:34:29.025 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi077.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-08-21T14:34:29.049 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:29 +0000 (0:00:00.278) 0:03:07.316 ******* 2021-08-21T14:34:29.771 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-08-21T14:34:29.772 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-08-21T14:34:29.802 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:29 +0000 (0:00:00.752) 0:03:08.068 ******* 2021-08-21T14:34:29.873 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-08-21T14:34:29.875 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi077"}, "changed": false} 2021-08-21T14:34:29.896 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:29 +0000 (0:00:00.094) 0:03:08.163 ******* 2021-08-21T14:34:30.941 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi077.front.sepia.ceph.com", "ansible_hostname": "smithi077", "ansible_nodename": "smithi077"}, "changed": true, "name": "smithi077"} 2021-08-21T14:34:30.963 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:30 +0000 (0:00:01.065) 0:03:09.228 ******* 2021-08-21T14:34:31.615 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-08-21T14:34:31.636 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:31 +0000 (0:00:00.674) 0:03:09.902 ******* 2021-08-21T14:34:32.146 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-08-21T14:34:32.148 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-08-21T14:34:32.171 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:32 +0000 (0:00:00.534) 0:03:10.437 ******* 2021-08-21T14:34:32.751 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi077.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-08-21T14:34:32.776 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:32 +0000 (0:00:00.604) 0:03:11.041 ******* 2021-08-21T14:34:33.336 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-08-21T14:34:33.363 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:33 +0000 (0:00:00.582) 0:03:11.625 ******* 2021-08-21T14:34:33.625 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-08-21T14:34:33.643 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:33 +0000 (0:00:00.284) 0:03:11.909 ******* 2021-08-21T14:34:33.875 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-08-21T14:34:33.897 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:33 +0000 (0:00:00.253) 0:03:12.163 ******* 2021-08-21T14:34:34.309 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* ok: [smithi077.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1254, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-08-21T14:34:34.329 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:34 +0000 (0:00:00.432) 0:03:12.595 ******* 2021-08-21T14:34:34.989 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-08-21T14:34:34.991 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-08-21T14:34:35.020 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:35 +0000 (0:00:00.687) 0:03:13.283 ******* 2021-08-21T14:34:35.688 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-08-21T14:34:35.711 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:35 +0000 (0:00:00.691) 0:03:13.975 ******* 2021-08-21T14:34:38.196 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.245147", "end": "2021-08-21 14:34:38.168076", "rc": 0, "start": "2021-08-21 14:34:35.922929", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:34:38.218 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:38 +0000 (0:00:02.509) 0:03:16.484 ******* 2021-08-21T14:34:38.291 INFO:teuthology.task.ansible.out: TASK [testnode : Clean up Stream distro-provided repos] ************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:38.310 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:38 +0000 (0:00:00.092) 0:03:16.577 ******* 2021-08-21T14:34:38.390 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-08-21T14:34:38.391 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:38.416 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:38 +0000 (0:00:00.105) 0:03:16.683 ******* 2021-08-21T14:34:38.499 INFO:teuthology.task.ansible.out: TASK [testnode : Include CentOS Stream specific variables] ********************* skipping: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml", "skip_reason": "Conditional result was False"} 2021-08-21T14:34:38.531 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:38 +0000 (0:00:00.115) 0:03:16.798 ******* 2021-08-21T14:34:39.072 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-08-21T14:34:39.074 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-08-21T14:34:39.508 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-08-21T14:34:39.952 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-08-21T14:34:40.392 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=CentOS-PowerTools-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "20107eac0ba9a84b318fa5f29b02a387051bff98", "dest": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-PowerTools-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-08-21T14:34:40.416 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:40 +0000 (0:00:01.885) 0:03:18.683 ******* 2021-08-21T14:34:40.928 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-08-21T14:34:40.930 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-08-21T14:34:40.951 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:40 +0000 (0:00:00.534) 0:03:19.218 ******* 2021-08-21T14:34:41.494 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-08-21T14:34:41.496 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-08-21T14:34:41.940 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-08-21T14:34:42.386 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-08-21T14:34:42.830 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'key': 'CentOS-PowerTools', 'value': {'name': 'CentOS-$releasever - PowerTools', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "76441d1028936932e965a46c12ad393f767201d3", "dest": "/etc/yum.repos.d/CentOS-PowerTools.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-PowerTools", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-PowerTools-mirrorlist", "name": "CentOS-$releasever - PowerTools"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-PowerTools.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 292, "state": "file", "uid": 0} 2021-08-21T14:34:42.854 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:42 +0000 (0:00:01.903) 0:03:21.121 ******* 2021-08-21T14:34:43.355 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-08-21T14:34:43.357 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item=ceph/python3-asyncssh) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ceph/python3-asyncssh"], "delta": "0:00:00.261424", "end": "2021-08-21 14:34:43.332144", "item": "ceph/python3-asyncssh", "rc": 0, "start": "2021-08-21 14:34:43.070720", "stderr": "Enabling a Copr repository. Please note that this repository is not part\nof the main distribution, and quality may vary.\n\nThe Fedora Project does not exercise any power over the contents of\nthis repository beyond the rules outlined in the Copr FAQ at\n,\nand packages are not held to any quality or security level.\n\nPlease do not file bug reports about these packages in Fedora\nBugzilla. In case of problems, contact the owner of this repository.", "stderr_lines": ["Enabling a Copr repository. Please note that this repository is not part", "of the main distribution, and quality may vary.", "", "The Fedora Project does not exercise any power over the contents of", "this repository beyond the rules outlined in the Copr FAQ at", ",", "and packages are not held to any quality or security level.", "", "Please do not file bug reports about these packages in Fedora", "Bugzilla. In case of problems, contact the owner of this repository."], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'comma 2021-08-21T14:34:43.359 INFO:teuthology.task.ansible.out:nd_warnings=False' in ansible.cfg to get rid of this message. 2021-08-21T14:34:43.380 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:43 +0000 (0:00:00.525) 0:03:21.647 ******* 2021-08-21T14:34:43.462 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:43.484 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:43 +0000 (0:00:00.104) 0:03:21.751 ******* 2021-08-21T14:34:43.577 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-08-21T14:34:43.580 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-08-21T14:34:43.602 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:43 +0000 (0:00:00.117) 0:03:21.868 ******* 2021-08-21T14:34:43.673 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-08-21T14:34:43.675 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:43.698 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:43 +0000 (0:00:00.095) 0:03:21.964 ******* 2021-08-21T14:34:43.778 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-08-21T14:34:43.801 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:43 +0000 (0:00:00.103) 0:03:22.067 ******* 2021-08-21T14:34:43.880 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-08-21T14:34:43.882 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:43.903 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:43 +0000 (0:00:00.102) 0:03:22.170 ******* 2021-08-21T14:34:43.989 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:44.017 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:44 +0000 (0:00:00.108) 0:03:22.278 ******* 2021-08-21T14:34:44.271 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* fatal: [smithi077.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006579", "end": "2021-08-21 14:34:44.244280", "msg": "non-zero return code", "rc": 1, "start": "2021-08-21 14:34:44.237701", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-08-21T14:34:44.296 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:44 +0000 (0:00:00.284) 0:03:22.562 ******* 2021-08-21T14:34:44.377 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:44.405 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:44 +0000 (0:00:00.104) 0:03:22.667 ******* 2021-08-21T14:34:48.745 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-08-21T14:34:48.769 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:48 +0000 (0:00:04.367) 0:03:27.035 ******* 2021-08-21T14:34:51.378 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-08-21T14:34:51.380 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-10.el8.x86_64", "Removed: boost-random-1.66.0-10.el8.x86_64"]} 2021-08-21T14:34:51.412 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:51 +0000 (0:00:02.632) 0:03:29.668 ******* 2021-08-21T14:34:58.596 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-08-21T14:34:58.600 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: qemu-kvm-block-rbd-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: qemu-img-15:4.2.0-48.module_el8.4.0+885+5e18b468.3.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-core-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-img-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64", "Removed: qemu-kvm-common-15:4.2.0-48.module_el8.4.0+783+f8734d30.x86_64"]} 2021-08-21T14:34:58.620 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:58 +0000 (0:00:07.218) 0:03:36.886 ******* 2021-08-21T14:34:58.702 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:34:58.724 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:34:58 +0000 (0:00:00.104) 0:03:36.991 ******* 2021-08-21T14:35:00.365 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:35:00.388 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:00 +0000 (0:00:01.663) 0:03:38.654 ******* 2021-08-21T14:35:02.010 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:35:02.035 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:02 +0000 (0:00:01.646) 0:03:40.301 ******* 2021-08-21T14:35:02.432 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveState": "inactive", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", 2021-08-21T14:35:02.435 INFO:teuthology.task.ansible.out: "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-08-21T14:35:02.453 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:02 +0000 (0:00:00.419) 0:03:40.720 ******* 2021-08-21T14:35:02.828 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi077.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-08-21T14:35:02.852 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:02 +0000 (0:00:00.398) 0:03:41.119 ******* 2021-08-21T14:35:03.498 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-08-21T14:35:03.524 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:03 +0000 (0:00:00.670) 0:03:41.789 ******* 2021-08-21T14:35:03.602 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:03.625 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:03 +0000 (0:00:00.101) 0:03:41.891 ******* 2021-08-21T14:35:03.718 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:03.741 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:03 +0000 (0:00:00.113) 0:03:42.005 ******* 2021-08-21T14:35:03.827 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:03.854 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:03 +0000 (0:00:00.108) 0:03:42.114 ******* 2021-08-21T14:35:03.931 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:03.970 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:03 +0000 (0:00:00.121) 0:03:42.235 ******* 2021-08-21T14:35:04.062 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:04.084 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.114) 0:03:42.350 ******* 2021-08-21T14:35:04.178 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:04.203 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.116) 0:03:42.466 ******* 2021-08-21T14:35:04.289 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-08-21T14:35:04.328 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.114) 0:03:42.581 ******* 2021-08-21T14:35:04.425 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:04.449 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.134) 0:03:42.715 ******* 2021-08-21T14:35:04.550 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:04.566 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.117) 0:03:42.832 ******* 2021-08-21T14:35:04.697 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:04.718 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.144) 0:03:42.976 ******* 2021-08-21T14:35:04.794 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:04.819 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.108) 0:03:43.086 ******* 2021-08-21T14:35:04.905 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-08-21T14:35:04.926 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:04 +0000 (0:00:00.106) 0:03:43.192 ******* 2021-08-21T14:35:05.002 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.059 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.129) 0:03:43.322 ******* 2021-08-21T14:35:05.137 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-08-21T14:35:05.147 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.174 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.112) 0:03:43.434 ******* 2021-08-21T14:35:05.267 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.292 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.121) 0:03:43.556 ******* 2021-08-21T14:35:05.385 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.403 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.113) 0:03:43.669 ******* 2021-08-21T14:35:05.484 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.512 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.109) 0:03:43.779 ******* 2021-08-21T14:35:05.611 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-08-21T14:35:05.612 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.635 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.123) 0:03:43.902 ******* 2021-08-21T14:35:05.746 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.766 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.129) 0:03:44.032 ******* 2021-08-21T14:35:05.860 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:05.904 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:05 +0000 (0:00:00.135) 0:03:44.167 ******* 2021-08-21T14:35:05.982 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.012 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.106) 0:03:44.274 ******* 2021-08-21T14:35:06.103 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.130 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.119) 0:03:44.393 ******* 2021-08-21T14:35:06.138 INFO:teuthology.task.ansible.out: 2021-08-21T14:35:06.255 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.281 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.150) 0:03:44.543 ******* 2021-08-21T14:35:06.425 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.455 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.173) 0:03:44.717 ******* 2021-08-21T14:35:06.589 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.631 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.176) 0:03:44.893 ******* 2021-08-21T14:35:06.742 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.771 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.142) 0:03:45.036 ******* 2021-08-21T14:35:06.904 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:06.943 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:06 +0000 (0:00:00.173) 0:03:45.209 ******* 2021-08-21T14:35:07.033 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.057 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.113) 0:03:45.323 ******* 2021-08-21T14:35:07.183 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.204 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.146) 0:03:45.470 ******* 2021-08-21T14:35:07.283 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-08-21T14:35:07.286 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.318 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.108) 0:03:45.579 ******* 2021-08-21T14:35:07.403 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.422 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.107) 0:03:45.687 ******* 2021-08-21T14:35:07.517 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.560 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.129) 0:03:45.816 ******* 2021-08-21T14:35:07.634 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-08-21T14:35:07.636 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.657 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.107) 0:03:45.923 ******* 2021-08-21T14:35:07.813 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.849 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.183) 0:03:46.107 ******* 2021-08-21T14:35:07.916 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-08-21T14:35:07.921 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:07.950 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:07 +0000 (0:00:00.105) 0:03:46.212 ******* 2021-08-21T14:35:08.059 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:08.085 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:08 +0000 (0:00:00.130) 0:03:46.343 ******* 2021-08-21T14:35:08.192 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:08.228 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:08 +0000 (0:00:00.147) 0:03:46.491 ******* 2021-08-21T14:35:08.369 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:08.407 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:08 +0000 (0:00:00.179) 0:03:46.670 ******* 2021-08-21T14:35:08.491 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:08.544 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:08 +0000 (0:00:00.131) 0:03:46.801 ******* 2021-08-21T14:35:08.644 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:08.684 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:08 +0000 (0:00:00.147) 0:03:46.949 ******* 2021-08-21T14:35:08.948 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-08-21T14:35:08.951 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-08-21T14:35:08.979 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:08 +0000 (0:00:00.296) 0:03:47.245 ******* 2021-08-21T14:35:09.259 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:35:09.276 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:09 +0000 (0:00:00.290) 0:03:47.535 ******* 2021-08-21T14:35:09.537 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-08-21T14:35:09.566 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:09 +0000 (0:00:00.292) 0:03:47.827 ******* 2021-08-21T14:35:09.718 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:09.746 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:09 +0000 (0:00:00.181) 0:03:48.009 ******* 2021-08-21T14:35:09.820 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:09.845 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:09 +0000 (0:00:00.103) 0:03:48.112 ******* 2021-08-21T14:35:09.942 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:09.966 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:09 +0000 (0:00:00.119) 0:03:48.231 ******* 2021-08-21T14:35:10.058 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.090 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.123) 0:03:48.355 ******* 2021-08-21T14:35:10.208 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.237 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.147) 0:03:48.503 ******* 2021-08-21T14:35:10.339 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.367 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.130) 0:03:48.633 ******* 2021-08-21T14:35:10.458 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.487 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.120) 0:03:48.753 ******* 2021-08-21T14:35:10.587 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.621 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.130) 0:03:48.884 ******* 2021-08-21T14:35:10.721 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.750 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.132) 0:03:49.016 ******* 2021-08-21T14:35:10.836 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:10.866 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.115) 0:03:49.131 ******* 2021-08-21T14:35:10.983 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.015 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:10 +0000 (0:00:00.139) 0:03:49.270 ******* 2021-08-21T14:35:11.095 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.126 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.111) 0:03:49.382 ******* 2021-08-21T14:35:11.198 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.225 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.107) 0:03:49.489 ******* 2021-08-21T14:35:11.322 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.339 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.115) 0:03:49.605 ******* 2021-08-21T14:35:11.416 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.445 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.105) 0:03:49.711 ******* 2021-08-21T14:35:11.536 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-08-21T14:35:11.539 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.573 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.128) 0:03:49.839 ******* 2021-08-21T14:35:11.671 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-08-21T14:35:11.673 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.705 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.132) 0:03:49.972 ******* 2021-08-21T14:35:11.803 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.822 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.116) 0:03:50.089 ******* 2021-08-21T14:35:11.906 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:11.934 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:11 +0000 (0:00:00.110) 0:03:50.199 ******* 2021-08-21T14:35:12.016 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-08-21T14:35:12.017 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.052 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:12 +0000 (0:00:00.118) 0:03:50.318 ******* 2021-08-21T14:35:12.132 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.179 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:12 +0000 (0:00:00.103) 0:03:50.421 ******* 2021-08-21T14:35:12.244 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.291 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:12 +0000 (0:00:00.135) 0:03:50.557 ******* 2021-08-21T14:35:12.402 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-08-21T14:35:12.437 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:12 +0000 (0:00:00.139) 0:03:50.697 ******* 2021-08-21T14:35:12.541 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi077.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.579 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi077.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.590 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.603 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.623 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.625 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:12.658 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:12 +0000 (0:00:00.226) 0:03:50.923 ******* 2021-08-21T14:35:13.093 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.006126", "end": "2021-08-21 14:35:13.051764", "rc": 0, "start": "2021-08-21 14:35:13.045638", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_321319b12ea4ff9b63c7655015a3156de2c3b279/virtualenv/bin/teuthology-lock"]} 2021-08-21T14:35:13.132 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:13 +0000 (0:00:00.470) 0:03:51.394 ******* 2021-08-21T14:35:13.285 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:13.315 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:13 +0000 (0:00:00.184) 0:03:51.578 ******* 2021-08-21T14:35:13.420 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:13.450 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:13 +0000 (0:00:00.136) 0:03:51.714 ******* 2021-08-21T14:35:13.549 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:13.590 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:13 +0000 (0:00:00.138) 0:03:51.853 ******* 2021-08-21T14:35:13.685 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:13.717 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:13 +0000 (0:00:00.125) 0:03:51.979 ******* 2021-08-21T14:35:15.341 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:35:15.367 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:15 +0000 (0:00:01.652) 0:03:53.631 ******* 2021-08-21T14:35:15.461 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-08-21T14:35:15.464 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259460096, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114126, 'block_used': 12980982, 'inode_total': 61054976, 'inode_available': 60985864, 'inode_used': 69112, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114126, "block_size": 4096, "block_total": 240095108, "block_used": 12980982, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985864, "inode_total": 61054976, "inode_used": 69112, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259460096, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}} 2021-08-21T14:35:15.492 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:15 +0000 (0:00:00.123) 0:03:53.755 ******* 2021-08-21T14:35:15.761 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-08-21T14:35:15.763 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005124", "end": "2021-08-21 14:35:15.732220", "rc": 0, "start": "2021-08-21 14:35:15.727096", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:15.789 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:15 +0000 (0:00:00.300) 0:03:54.055 ******* 2021-08-21T14:35:15.887 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi077.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259460096, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114126, 'block_used': 12980982, 'inode_total': 61054976, 'inode_available': 60985864, 'inode_used': 69112, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114126, "block_size": 4096, "block_total": 240095108, "block_used": 12980982, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985864, "inode_total": 61054976, "inode_used": 69112, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259460096, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:15.912 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:15 +0000 (0:00:00.119) 0:03:54.175 ******* 2021-08-21T14:35:15.986 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-08-21T14:35:16.009 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:16 +0000 (0:00:00.099) 0:03:54.275 ******* 2021-08-21T14:35:17.664 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-08-21T14:35:17.666 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8HE7RD9Iy2wP8RZFXA1qOAgboFngl05f'], 'uuids': ['bcedd88d-b02e-403e-a821-e81578d6f8b5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.395393", "end": "2021-08-21 14:35:17.631319", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8HE7RD9Iy2wP8RZFXA1qOAgboFngl05f"], "labels": [], "masters": [], "uuids": ["bcedd88d-b02e-403e-a821-e81578d6f8b5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:16.235926", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:35:18.938 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-D7wtjM-qjga-Wxe4-SatP-Ij1w-OiMY-IYPnxU', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN', 'nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.064723", "end": "2021-08-21 14:35:18.892040", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-D7wtjM-qjga-Wxe4-SatP-Ij1w-OiMY-IYPnxU", "nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN", "nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:17.827317", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-08-21T14:35:18.944 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:35:20.198 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIAMqWmGn6WwaafojZo4aMMMBUKCc3bxE9'], 'uuids': ['6d28f57c-b74b-4c95-8a30-e4cd8072c721'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.060656", "end": "2021-08-21 14:35:20.161941", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIAMqWmGn6WwaafojZo4aMMMBUKCc3bxE9"], "labels": [], "masters": [], "uuids": ["6d28f57c-b74b-4c95-8a30-e4cd8072c721"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:19.101285", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:35:21.450 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8CqPuVCzVCvQ3ISIpCLXcmT2wTzaoQyd'], 'uuids': ['e4e6712c-44b1-4926-b37e-14f1cfc6857f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.040814", "end": "2021-08-21 14:35:21.418571", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8CqPuVCzVCvQ3ISIpCLXcmT2wTzaoQyd"], "labels": [], "masters": [], "uuids": ["e4e6712c-44b1-4926-b37e-14f1cfc6857f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:20.377757", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:35:22.695 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI13SZNEycUQMBpyoMNIrngdof0oNoNU6a'], 'uuids': ['d719e95c-d970-49b8-970a-c4cb6d0eb5b1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.054146", "end": "2021-08-21 14:35:22.662182", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI13SZNEycUQMBpyoMNIrngdof0oNoNU6a"], "labels": [], "masters": [], "uuids": ["d719e95c-d970-49b8-970a-c4cb6d0eb5b1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:21.608036", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:35:22.698 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJQN', 'scsi-35000c500918d87b8', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN', 'wwn-0x5000c500918d87b8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5LJQN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918d87b8', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJQN-part1', 'scsi-35000c500918d87b8-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN-part1', 'wwn-0x5000c500918d87b8-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJQN", "scsi-35000c500918d87b8", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN", "wwn-0x5000c500918d87b8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-08-21T14:35:22.700 INFO:teuthology.task.ansible.out:Z1W5LJQN-part1", "scsi-35000c500918d87b8-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN-part1", "wwn-0x5000c500918d87b8-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJQN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918d87b8"}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:23.950 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIaRsbIfBTt6ExzX873AnJIC0su7Ueu1Jn'], 'uuids': ['e5bcdf6c-2316-466a-a5d0-a601b3a11723'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.049281", "end": "2021-08-21 14:35:23.911781", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIaRsbIfBTt6ExzX873AnJIC0su7Ueu1Jn"], "labels": [], "masters": [], "uuids": ["e5bcdf6c-2316-466a-a5d0-a601b3a11723"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:22.862500", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-08-21T14:35:23.978 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:23 +0000 (0:00:07.969) 0:04:02.244 ******* 2021-08-21T14:35:24.427 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8HE7RD9Iy2wP8RZFXA1qOAgboFngl05f'], 'uuids': ['bcedd88d-b02e-403e-a821-e81578d6f8b5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.140839", "end": "2021-08-21 14:35:24.391535", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8HE7RD9Iy2wP8RZFXA1qOAgboFngl05f"], "labels": [], "masters": [], "uuids": ["bcedd88d-b02e-403e-a821-e81578d6f8b5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:24.250696", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138114 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138114 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:24.693 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-D7wtjM-qjga-Wxe4-SatP-Ij1w-OiMY-IYPnxU', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN', 'nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.040183", "end": "2021-08-21 14:35:24.641101", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-D7wtjM-qjga-Wxe4-SatP-Ij1w-OiMY-IYPnxU", "nvme-INTEL_SSDPEDMD400G4_PHFT620400ZJ400BGN", "nvme-nvme.8086-504846543632303430305a4a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:24.600918", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0374447 s, 3.1 GB/s", "std 2021-08-21T14:35:24.696 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0374447 s, 3.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:25.039 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIAMqWmGn6WwaafojZo4aMMMBUKCc3bxE9'], 'uuids': ['6d28f57c-b74b-4c95-8a30-e4cd8072c721'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.151565", "end": "2021-08-21 14:35:25.008247", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIAMqWmGn6WwaafojZo4aMMMBUKCc3bxE9"], "labels": [], "masters": [], "uuids": ["6d28f57c-b74b-4c95-8a30-e4cd8072c721"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:24.856682", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148732 s, 776 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148732 s, 776 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:25.373 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8CqPuVCzVCvQ3ISIpCLXcmT2wTzaoQyd'], 'uuids': ['e4e6712c-44b1-4926-b37e-14f1cfc6857f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.143466", "end": "2021-08-21 14:35:25.336472", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI8CqPuVCzVCvQ3ISIpCLXcmT2wTzaoQyd"], "labels": [], "masters": [], "uuids": ["e4e6712c-44b1-4926-b37e-14f1cfc6857f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:25.193006", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140845 s, 819 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140845 s, 819 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:25.755 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI13SZNEycUQMBpyoMNIrngdof0oNoNU6a'], 'uuids': ['d719e95c-d970-49b8-970a-c4cb6d0eb5b1'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.144953", "end": "2021-08-21 14:35:25.710847", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzI13SZNEycUQMBpyoMNIrngdof0oNoNU6a"], "labels": [], "masters": [], "uuids": ["d719e95c-d970-49b8-970a-c4cb6d0eb5b1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:25.565894", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142392 s, 810 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142392 s, 810 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:25.769 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJQN', 'scsi-35000c500918d87b8', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN', 'wwn-0x5000c500918d87b8'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5LJQN', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c500918d87b8', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJQN-part1', 'scsi-35000c500918d87b8-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN-part1', 'wwn-0x5000c500918d87b8-part1'], 'uuids': ['fcc12580-8f3b-4cec-aef7-e7427fa239b7'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5LJQN", "scsi-35000c500918d87b8", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN", "wwn-0x5000c500918d87b8"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5LJQN-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5LJQN-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-08-21T14:35:25.774 INFO:teuthology.task.ansible.out:Z1W5LJQN-part1", "scsi-35000c500918d87b8-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5LJQN-part1", "wwn-0x5000c500918d87b8-part1"], "labels": [], "masters": [], "uuids": ["fcc12580-8f3b-4cec-aef7-e7427fa239b7"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5LJQN", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500918d87b8"}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:26.195 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIaRsbIfBTt6ExzX873AnJIC0su7Ueu1Jn'], 'uuids': ['e5bcdf6c-2316-466a-a5d0-a601b3a11723'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.143809", "end": "2021-08-21 14:35:26.170305", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-oKC9n3ovtNazt4sEprWY6kBMndGyAyzIaRsbIfBTt6ExzX873AnJIC0su7Ueu1Jn"], "labels": [], "masters": [], "uuids": ["e5bcdf6c-2316-466a-a5d0-a601b3a11723"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-08-21 14:35:26.026496", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.141118 s, 817 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.141118 s, 817 MB/s"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:26.267 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:26 +0000 (0:00:02.287) 0:04:04.532 ******* 2021-08-21T14:35:26.639 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi077.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.048095", "end": "2021-08-21 14:35:26.503957", "rc": 0, "start": "2021-08-21 14:35:26.455862", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:26.665 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:26 +0000 (0:00:00.391) 0:04:04.923 ******* 2021-08-21T14:35:26.936 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-08-21T14:35:26.958 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.039497", "end": "2021-08-21 14:35:26.909872", "rc": 0, "start": "2021-08-21 14:35:26.870375", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:26.986 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:26 +0000 (0:00:00.317) 0:04:05.241 ******* 2021-08-21T14:35:27.167 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-08-21T14:35:27.280 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:27 +0000 (0:00:00.305) 0:04:05.546 ******* 2021-08-21T14:35:27.334 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:27.345 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:27 +0000 (0:00:00.063) 0:04:05.610 ******* 2021-08-21T14:35:27.462 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:27.482 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:27 +0000 (0:00:00.137) 0:04:05.748 ******* 2021-08-21T14:35:27.558 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:27.581 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:27 +0000 (0:00:00.093) 0:04:05.841 ******* 2021-08-21T14:35:27.654 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:27.684 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:27 +0000 (0:00:00.101) 0:04:05.943 ******* 2021-08-21T14:35:27.938 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-08-21T14:35:27.974 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:27 +0000 (0:00:00.288) 0:04:06.231 ******* 2021-08-21T14:35:28.054 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi077.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930259460096, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227114126, 'block_used': 12980982, 'inode_total': 61054976, 'inode_available': 60985864, 'inode_used': 69112, 'uuid': 'fcc12580-8f3b-4cec-aef7-e7427fa239b7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227114126, "block_size": 4096, "block_total": 240095108, "block_used": 12980982, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60985864, "inode_total": 61054976, "inode_used": 69112, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930259460096, "size_total": 983429562368, "uuid": "fcc12580-8f3b-4cec-aef7-e7427fa239b7"}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.084 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.116) 0:04:06.348 ******* 2021-08-21T14:35:28.151 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.186 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.095) 0:04:06.444 ******* 2021-08-21T14:35:28.244 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.269 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.090) 0:04:06.534 ******* 2021-08-21T14:35:28.371 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.434 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.156) 0:04:06.690 ******* 2021-08-21T14:35:28.526 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.598 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.173) 0:04:06.864 ******* 2021-08-21T14:35:28.658 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-08-21T14:35:28.678 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.705 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.089) 0:04:06.953 ******* 2021-08-21T14:35:28.769 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:28.810 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:28 +0000 (0:00:00.122) 0:04:07.075 ******* 2021-08-21T14:35:29.461 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-08-21T14:35:29.504 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:29 +0000 (0:00:00.690) 0:04:07.765 ******* 2021-08-21T14:35:30.333 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-08-21T14:35:30.347 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:35:30.755 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:35:31.110 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:35:31.490 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:35:31.931 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-08-21T14:35:31.984 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:31 +0000 (0:00:02.456) 0:04:10.222 ******* 2021-08-21T14:35:32.226 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-08-21T14:35:32.251 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:32 +0000 (0:00:00.294) 0:04:10.516 ******* 2021-08-21T14:35:32.595 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:35:32.798 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:35:33.030 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:35:33.220 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-08-21T14:35:33.275 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:33.305 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:33 +0000 (0:00:01.043) 0:04:11.560 ******* 2021-08-21T14:35:33.386 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:33.424 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:33 +0000 (0:00:00.114) 0:04:11.675 ******* 2021-08-21T14:35:33.464 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-08-21T14:35:33.466 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:33.493 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:33 +0000 (0:00:00.079) 0:04:11.755 ******* 2021-08-21T14:35:33.759 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:35:33.790 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:33 +0000 (0:00:00.297) 0:04:12.053 ******* 2021-08-21T14:35:33.842 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-08-21T14:35:33.882 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:33 +0000 (0:00:00.095) 0:04:12.148 ******* 2021-08-21T14:35:35.623 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi077.front.sepia.ceph.com] => {"changed": true} 2021-08-21T14:35:35.650 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:35 +0000 (0:00:01.767) 0:04:13.915 ******* 2021-08-21T14:35:36.398 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-08-21T14:35:36.422 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.769) 0:04:14.685 ******* 2021-08-21T14:35:36.481 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.500 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.079) 0:04:14.764 ******* 2021-08-21T14:35:36.582 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.598 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.099) 0:04:14.864 ******* 2021-08-21T14:35:36.638 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.673 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.065) 0:04:14.929 ******* 2021-08-21T14:35:36.694 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.720 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.056) 0:04:14.986 ******* 2021-08-21T14:35:36.756 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.782 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.062) 0:04:15.048 ******* 2021-08-21T14:35:36.840 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.866 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.081) 0:04:15.129 ******* 2021-08-21T14:35:36.957 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:36.986 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:36 +0000 (0:00:00.117) 0:04:15.247 ******* 2021-08-21T14:35:37.046 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.077 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.096) 0:04:15.344 ******* 2021-08-21T14:35:37.136 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.182 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.095) 0:04:15.440 ******* 2021-08-21T14:35:37.232 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.258 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.083) 0:04:15.523 ******* 2021-08-21T14:35:37.315 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.338 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.081) 0:04:15.605 ******* 2021-08-21T14:35:37.397 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.418 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.079) 0:04:15.685 ******* 2021-08-21T14:35:37.466 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.498 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.073) 0:04:15.759 ******* 2021-08-21T14:35:37.575 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.596 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.100) 0:04:15.860 ******* 2021-08-21T14:35:37.603 INFO:teuthology.task.ansible.out: 2021-08-21T14:35:37.634 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:37.657 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:37 +0000 (0:00:00.063) 0:04:15.924 ******* 2021-08-21T14:35:38.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-08-21T14:35:38.235 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:38 +0000 (0:00:00.577) 0:04:16.501 ******* 2021-08-21T14:35:38.237 INFO:teuthology.task.ansible.out: 2021-08-21T14:35:38.609 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-08-21T14:35:38.614 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveState": "active", "AllowedCPUs": "", "AllowedMemoryNodes": "", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "Delegate": "no", "DevicePolicy": "auto", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "802", "ExecMainStartTimestamp": "Sat 2021-08-21 14:30:41 UTC", "ExecMainStartTimestampMonotonic": "11931985", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FileDescriptorStoreMax": "0", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSI 2021-08-21T14:35:38.616 INFO:teuthology.task.ansible.out:ZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127245", "LimitNPROCSoft": "127245", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127245", "LimitSIGPENDINGSoft": "127245", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "MainPID": "802", "MemoryAccounting": "yes", "MemoryCurrent": "4296704", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemoryMin": "0", "MemorySwapMax": "infinity", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Nice": "0", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "RemainAfterExit": "no", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeMaxUSec": "infinity", "SecureBits": "0", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StatusErrno": "0", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203592", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Type": "forking", "UID": "[not set]", "UMask": "0022", "WatchdogTimestamp": "Sat 2021-08-21 1 2021-08-21T14:35:38.626 INFO:teuthology.task.ansible.out:4:30:41 UTC", "WatchdogTimestampMonotonic": "11931995", "WatchdogUSec": "0"}} 2021-08-21T14:35:38.660 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:38 +0000 (0:00:00.424) 0:04:16.926 ******* 2021-08-21T14:35:38.711 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-08-21T14:35:38.746 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:38 +0000 (0:00:00.084) 0:04:17.010 ******* 2021-08-21T14:35:38.823 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-08-21T14:35:38.861 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:38 +0000 (0:00:00.116) 0:04:17.127 ******* 2021-08-21T14:35:38.917 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:38.949 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:38 +0000 (0:00:00.088) 0:04:17.216 ******* 2021-08-21T14:35:38.992 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:39.017 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:39 +0000 (0:00:00.065) 0:04:17.282 ******* 2021-08-21T14:35:39.046 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:39.079 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:39 +0000 (0:00:00.056) 0:04:17.338 ******* 2021-08-21T14:35:39.290 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-08-21T14:35:39.319 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:39 +0000 (0:00:00.246) 0:04:17.585 ******* 2021-08-21T14:35:39.940 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1254, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-08-21T14:35:39.975 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:39 +0000 (0:00:00.653) 0:04:18.238 ******* 2021-08-21T14:35:41.502 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-08-21T14:35:41.538 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:41 +0000 (0:00:01.563) 0:04:19.802 ******* 2021-08-21T14:35:41.590 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:41.630 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:41 +0000 (0:00:00.091) 0:04:19.894 ******* 2021-08-21T14:35:41.658 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-08-21T14:35:41.682 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:41 +0000 (0:00:00.053) 0:04:19.948 ******* 2021-08-21T14:35:41.913 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-08-21T14:35:41.936 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:41 +0000 (0:00:00.254) 0:04:20.203 ******* 2021-08-21T14:35:41.993 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:42.022 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.080) 0:04:20.284 ******* 2021-08-21T14:35:42.068 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-08-21T14:35:42.086 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.067) 0:04:20.351 ******* 2021-08-21T14:35:42.142 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:42.187 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.097) 0:04:20.449 ******* 2021-08-21T14:35:42.425 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-08-21T14:35:42.456 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.272) 0:04:20.722 ******* 2021-08-21T14:35:42.686 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi077.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-08-21T14:35:42.720 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.259) 0:04:20.981 ******* 2021-08-21T14:35:42.797 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:42.836 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.118) 0:04:21.100 ******* 2021-08-21T14:35:42.930 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi077.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:42.956 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.020 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:42 +0000 (0:00:00.160) 0:04:21.260 ******* 2021-08-21T14:35:43.082 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.150 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi077.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.196 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.187) 0:04:21.448 ******* 2021-08-21T14:35:43.270 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.279 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.097) 0:04:21.545 ******* 2021-08-21T14:35:43.320 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.346 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.065) 0:04:21.611 ******* 2021-08-21T14:35:43.395 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-08-21T14:35:43.405 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.436 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.087) 0:04:21.699 ******* 2021-08-21T14:35:43.483 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.528 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.096) 0:04:21.795 ******* 2021-08-21T14:35:43.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.601 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.072) 0:04:21.868 ******* 2021-08-21T14:35:43.656 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.699 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.095) 0:04:21.963 ******* 2021-08-21T14:35:43.732 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.771 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.067) 0:04:22.031 ******* 2021-08-21T14:35:43.825 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:43.856 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:43 +0000 (0:00:00.081) 0:04:22.113 ******* 2021-08-21T14:35:44.114 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi077.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005000", "end": "2021-08-21 14:35:44.061915", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-08-21 14:35:44.056915", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:44.365 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003160", "end": "2021-08-21 14:35:44.302388", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-08-21 14:35:44.299228", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:44.583 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002701", "end": "2021-08-21 14:35:44.541766", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-08-21 14:35:44.539065", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:44.790 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003064", "end": "2021-08-21 14:35:44.754361", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-08-21 14:35:44.751297", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:44.834 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:44 +0000 (0:00:00.972) 0:04:23.085 ******* 2021-08-21T14:35:44.877 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-08-21T14:35:44.905 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:44 +0000 (0:00:00.085) 0:04:23.171 ******* 2021-08-21T14:35:44.949 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:45.068 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-08-21T14:35:45.144 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:45 +0000 (0:00:00.229) 0:04:23.400 ******* 2021-08-21T14:35:45.927 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi077.front.sepia.ceph.com] 2021-08-21T14:35:46.013 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:46 +0000 (0:00:00.871) 0:04:24.271 ******* 2021-08-21T14:35:46.056 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-08-21T14:35:46.061 INFO:teuthology.task.ansible.out:skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:46.092 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:46 +0000 (0:00:00.085) 0:04:24.357 ******* 2021-08-21T14:35:46.193 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-08-21T14:35:46.214 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:46 +0000 (0:00:00.122) 0:04:24.480 ******* 2021-08-21T14:35:46.342 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-08-21T14:35:46.350 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-08-21T14:35:46.364 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:46 +0000 (0:00:00.151) 0:04:24.631 ******* 2021-08-21T14:35:52.050 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-catatonit-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Installed: podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64", "Installed: podman-docker-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.noarch", "Removed: podman-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-catatonit-3.0.1-7.module_el8.4.0+830+8027e1c4.x86_64", "Removed: podman-docker-3.0.1-7.module_el8.4.0+830+8027e1c4.noarch"]} 2021-08-21T14:35:52.088 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:52 +0000 (0:00:05.717) 0:04:30.349 ******* 2021-08-21T14:35:52.211 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** ok: [smithi077.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} Saturday 21 August 2021 14:35:52 +0000 (0:00:00.116) 0:04:30.465 ******* 2021-08-21T14:35:52.265 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:52.322 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:52 +0000 (0:00:00.105) 0:04:30.571 ******* 2021-08-21T14:35:52.655 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* ok: [smithi077.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:35:52.908 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-08-21T14:35:52.939 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:52 +0000 (0:00:00.630) 0:04:31.201 ******* 2021-08-21T14:35:53.639 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** ok: [smithi077.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-08-21T14:35:54.232 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": false, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0644", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "state": "file", "uid": 0} 2021-08-21T14:35:54.250 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:54 +0000 (0:00:01.307) 0:04:32.509 ******* 2021-08-21T14:35:55.928 INFO:teuthology.task.ansible.out: TASK [container-host : Ensure git is installed] ******************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-08-21T14:35:55.937 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:55 +0000 (0:00:01.690) 0:04:34.200 ******* 2021-08-21T14:35:58.357 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** ok: [smithi077.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-4b3zy5b8-build\nRequirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nRequirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)\nInstalling collected packages: registries-conf-ctl\n Found existing installation: registries-conf-ctl 1.0\n Uninstalling registries-conf-ctl-1.0:\n Successfully uninstalled registries-conf-ctl-1.0\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed registries-conf-ctl-1.0\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-4b3zy5b8-build", "Requirement already up-to-date: toml in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Requirement already up-to-date: docopt in /usr/local/lib/python3.6/site-packages (from registries-conf-ctl==1.0)", "Installing collected packages: registries-conf-ctl", " Found existing installation: registries-conf-ctl 1.0", " Uninstalling registries-conf-ctl-1.0 2021-08-21T14:35:58.360 INFO:teuthology.task.ansible.out::", " Successfully uninstalled registries-conf-ctl-1.0", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed registries-conf-ctl-1.0"], "version": null, "virtualenv": null} 2021-08-21T14:35:58.388 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:58 +0000 (0:00:02.448) 0:04:36.648 ******* 2021-08-21T14:35:58.473 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:58.530 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:58 +0000 (0:00:00.121) 0:04:36.769 ******* 2021-08-21T14:35:58.561 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:58.602 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:58 +0000 (0:00:00.090) 0:04:36.860 ******* 2021-08-21T14:35:59.058 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.155297", "end": "2021-08-21 14:35:59.020705", "rc": 0, "start": "2021-08-21 14:35:58.865408", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-08-21T14:35:59.092 INFO:teuthology.task.ansible.out:Saturday 21 August 2021 14:35:59 +0000 (0:00:00.496) 0:04:37.356 ******* 2021-08-21T14:35:59.168 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* skipping: [smithi077.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-08-21T14:35:59.250 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-08-21T14:35:59.285 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY [testnodes] *************************************************************** 2021-08-21T14:35:59.339 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* Saturday 21 August 2021 14:35:59 +0000 (0:00:00.247) 0:04:37.604 ******* 2021-08-21T14:36:00.155 INFO:teuthology.task.ansible.out:ok: [smithi077.front.sepia.ceph.com] 2021-08-21T14:36:00.196 INFO:teuthology.task.ansible.out: TASK [Touch /ceph-qa-ready] **************************************************** Saturday 21 August 2021 14:36:00 +0000 (0:00:00.839) 0:04:38.444 ******* 2021-08-21T14:36:00.490 INFO:teuthology.task.ansible.out:changed: [smithi077.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} PLAY RECAP ********************************************************************* smithi077.front.sepia.ceph.com : ok=132 changed=39 unreachable=0 failed=0 skipped=188 rescued=0 ignored=3 Saturday 21 August 2021 14:36:00 +0000 (0:00:00.300) 0:04:38.744 ******* =============================================================================== common : nrpe - Ensure SELinux policy is up to date -------------------- 37.10s Create all admin users with sudo access. ------------------------------- 35.79s users : Update authorized_keys using the keys repo --------------------- 32.82s common : nrpe - Load SELinux policy package ---------------------------- 15.14s common : nrpe - Remove SELinux policy package -------------------------- 14.95s Remove revoked users ---------------------------------------------------- 9.77s testnode : Zap all non-root disks --------------------------------------- 7.97s testnode : Install packages --------------------------------------------- 7.22s container-host : Install container packages ----------------------------- 5.72s testnode : Ensure ceph packages are not present. ------------------------ 4.37s common : Install Kerberos Packages (RedHat) ----------------------------- 3.95s testnode : Ensure ceph dependency packages are not present. ------------- 2.63s Gathering Facts --------------------------------------------------------- 2.61s testnode : Rebuild rpmdb ------------------------------------------------ 2.51s testnode : Create logical volume(s) ------------------------------------- 2.46s container-host : Install registries-conf-ctl ---------------------------- 2.45s testnode : Blow away lingering OSD data and FSIDs ----------------------- 2.29s testnod 2021-08-21T14:36:00.496 INFO:teuthology.task.ansible.out:e : Configure version specific repos in /etc/yum.repos.d/ -------- 1.90s users : Update authorized_keys for each user with literal keys ---------- 1.89s testnode : Configure local mirrorlists ---------------------------------- 1.89s 2021-08-21T14:36:01.141 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2021-08-21T14:36:01.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-08-21T14:36:01.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-08-21T14:36:01.693 DEBUG:teuthology.orchestra.run.smithi077:> true 2021-08-21T14:36:01.782 INFO:teuthology.run_tasks:Running task clock... 2021-08-21T14:36:01.910 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-08-21T14:36:01.919 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-08-21T14:36:01.926 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-08-21T14:36:02.050 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-08-21T14:36:02.051 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-08-21T14:36:02.116 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: ntpd: command not found 2021-08-21T14:36:02.118 INFO:teuthology.orchestra.run.smithi077.stdout:506 Cannot talk to daemon 2021-08-21T14:36:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-08-21T14:36:02.119 INFO:teuthology.orchestra.run.smithi077.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-08-21T14:36:02.166 INFO:teuthology.orchestra.run.smithi077.stderr:bash: ntpq: command not found 2021-08-21T14:36:02.233 INFO:teuthology.orchestra.run.smithi077.stdout:210 Number of sources = 4 2021-08-21T14:36:02.233 INFO:teuthology.orchestra.run.smithi077.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-08-21T14:36:02.234 INFO:teuthology.orchestra.run.smithi077.stdout:=============================================================================== 2021-08-21T14:36:02.260 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-08-21T14:36:02.275 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-08-21T14:36:02.275 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-08-21T14:36:02.275 INFO:teuthology.orchestra.run.smithi077.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-08-21T14:36:02.276 INFO:teuthology.run_tasks:Running task install... 2021-08-21T14:36:02.310 DEBUG:teuthology.task.install:project ceph 2021-08-21T14:36:02.384 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'}} 2021-08-21T14:36:02.391 DEBUG:teuthology.task.install:config {'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6'} 2021-08-21T14:36:02.391 INFO:teuthology.task.install:Using flavor: basic 2021-08-21T14:36:02.453 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-08-21T14:36:02.454 INFO:teuthology.task.install:extra packages: [] 2021-08-21T14:36:02.526 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2021-08-21T14:36:02.563 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS 2021-08-21T14:36:02.565 DEBUG:teuthology.misc:System to be installed: CentOS 2021-08-21T14:36:02.565 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'cae1ffd55a16af4627c36508fde05bf24312fad6', 'tag': None, 'wait_for_package': False} 2021-08-21T14:36:02.566 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2021-08-21T14:36:02.659 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS 2021-08-21T14:36:02.660 DEBUG:teuthology.misc:System to be installed: CentOS 2021-08-21T14:36:02.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:36:03.124 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/centos/8/flavors/default/ 2021-08-21T14:36:03.124 INFO:teuthology.task.install.rpm:Package version is 17.0.0-7135.gcae1ffd5 2021-08-21T14:36:03.449 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri-master-8.20.21/cae1ffd55a16af4627c36508fde05bf24312fad6/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-08-21T14:36:03.450 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-08-21T14:36:03.450 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-08-21T14:36:03.552 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-08-21T14:36:03.552 DEBUG:teuthology.orchestra.run.smithi077:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/cae1ffd55a16af4627c36508fde05bf24312fad6/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-08-21T14:36:03.714 DEBUG:teuthology.orchestra.run.smithi077:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-08-21T14:36:04.005 DEBUG:teuthology.orchestra.run.smithi077:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-08-21T14:36:04.216 INFO:teuthology.orchestra.run.smithi077.stdout:check_obsoletes = 1 2021-08-21T14:36:04.216 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum clean all 2021-08-21T14:36:04.450 INFO:teuthology.orchestra.run.smithi077.stdout:84 files removed 2021-08-21T14:36:04.469 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-radosgw 2021-08-21T14:36:05.043 INFO:teuthology.orchestra.run.smithi077.stdout:ceph packages for x86_64 327 kB/s | 76 kB 00:00 2021-08-21T14:36:05.268 INFO:teuthology.orchestra.run.smithi077.stdout:ceph noarch packages 40 kB/s | 7.3 kB 00:00 2021-08-21T14:36:05.486 INFO:teuthology.orchestra.run.smithi077.stdout:ceph source packages 7.5 kB/s | 1.4 kB 00:00 2021-08-21T14:36:05.811 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - AppStream 28 MB/s | 8.8 MB 00:00 2021-08-21T14:36:07.328 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - Base 45 MB/s | 5.6 MB 00:00 2021-08-21T14:36:08.179 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - Extras 162 kB/s | 10 kB 00:00 2021-08-21T14:36:08.990 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Linux 8 - AppStream 12 MB/s | 8.8 MB 00:00 2021-08-21T14:36:12.471 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Linux 8 - BaseOS 2.7 MB/s | 5.6 MB 00:02 2021-08-21T14:36:13.516 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS Linux 8 - Extras 38 kB/s | 10 kB 00:00 2021-08-21T14:36:13.656 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS-8 - PowerTools 24 MB/s | 2.3 MB 00:00 2021-08-21T14:36:15.806 INFO:teuthology.orchestra.run.smithi077.stdout:Copr repo for python3-asyncssh owned by ceph 2.0 kB/s | 3.5 kB 00:01 2021-08-21T14:36:17.491 INFO:teuthology.orchestra.run.smithi077.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 10 MB 00:01 2021-08-21T14:36:19.625 INFO:teuthology.orchestra.run.smithi077.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-08-21T14:36:21.814 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:36:21.864 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:36:21.865 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:36:21.865 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:36:21.870 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:36:21.870 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 8.5 M 2021-08-21T14:36:21.870 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrading: 2021-08-21T14:36:21.871 INFO:teuthology.orchestra.run.smithi077.stdout: librados2 x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 3.4 M 2021-08-21T14:36:21.871 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1 x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 3.9 M 2021-08-21T14:36:21.871 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:36:21.871 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 5.2 M 2021-08-21T14:36:21.872 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 20 M 2021-08-21T14:36:21.891 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 24 k 2021-08-21T14:36:21.904 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2021-08-21T14:36:21.904 INFO:teuthology.orchestra.run.smithi077.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-08-21T14:36:21.904 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2 x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 782 k 2021-08-21T14:36:21.904 INFO:teuthology.orchestra.run.smithi077.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-08-21T14:36:21.905 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq x86_64 0.9.0-3.el8 CentOS-Base 47 k 2021-08-21T14:36:21.905 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1 x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 482 k 2021-08-21T14:36:21.905 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-08-21T14:36:21.905 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2 x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 3.7 M 2021-08-21T14:36:21.905 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-08-21T14:36:21.906 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-08-21T14:36:21.906 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse 2021-08-21T14:36:21.906 INFO:teuthology.orchestra.run.smithi077.stdout: x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 45 k 2021-08-21T14:36:21.937 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 78 k 2021-08-21T14:36:21.937 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 178 k 2021-08-21T14:36:21.937 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-08-21T14:36:21.944 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 365 k 2021-08-21T14:36:21.945 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 346 k 2021-08-21T14:36:21.945 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 109 k 2021-08-21T14:36:21.945 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:21.945 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:36:21.946 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:36:21.946 INFO:teuthology.orchestra.run.smithi077.stdout:Install 21 Packages 2021-08-21T14:36:21.946 INFO:teuthology.orchestra.run.smithi077.stdout:Upgrade 2 Packages 2021-08-21T14:36:21.946 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:21.947 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 48 M 2021-08-21T14:36:21.947 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:36:22.754 INFO:teuthology.orchestra.run.smithi077.stdout:(1/23): ceph-base-17.0.0-7135.gcae1ffd5.el8.x86 5.6 MB/s | 5.2 MB 00:00 2021-08-21T14:36:22.774 INFO:teuthology.orchestra.run.smithi077.stdout:(2/23): ceph-radosgw-17.0.0-7135.gcae1ffd5.el8. 8.9 MB/s | 8.5 MB 00:00 2021-08-21T14:36:22.831 INFO:teuthology.orchestra.run.smithi077.stdout:(3/23): ceph-selinux-17.0.0-7135.gcae1ffd5.el8. 328 kB/s | 24 kB 00:00 2021-08-21T14:36:22.865 INFO:teuthology.orchestra.run.smithi077.stdout:(4/23): libcephfs2-17.0.0-7135.gcae1ffd5.el8.x8 8.7 MB/s | 782 kB 00:00 2021-08-21T14:36:22.886 INFO:teuthology.orchestra.run.smithi077.stdout:(5/23): libradosstriper1-17.0.0-7135.gcae1ffd5. 8.1 MB/s | 482 kB 00:00 2021-08-21T14:36:22.942 INFO:teuthology.orchestra.run.smithi077.stdout:(6/23): python3-ceph-argparse-17.0.0-7135.gcae1 813 kB/s | 45 kB 00:00 2021-08-21T14:36:22.999 INFO:teuthology.orchestra.run.smithi077.stdout:(7/23): python3-ceph-common-17.0.0-7135.gcae1ff 1.3 MB/s | 78 kB 00:00 2021-08-21T14:36:23.058 INFO:teuthology.orchestra.run.smithi077.stdout:(8/23): python3-cephfs-17.0.0-7135.gcae1ffd5.el 2.9 MB/s | 178 kB 00:00 2021-08-21T14:36:23.127 INFO:teuthology.orchestra.run.smithi077.stdout:(9/23): python3-rados-17.0.0-7135.gcae1ffd5.el8 5.2 MB/s | 365 kB 00:00 2021-08-21T14:36:23.203 INFO:teuthology.orchestra.run.smithi077.stdout:(10/23): python3-rbd-17.0.0-7135.gcae1ffd5.el8. 5.0 MB/s | 346 kB 00:00 2021-08-21T14:36:23.234 INFO:teuthology.orchestra.run.smithi077.stdout:(11/23): librgw2-17.0.0-7135.gcae1ffd5.el8.x86_ 11 MB/s | 3.7 MB 00:00 2021-08-21T14:36:23.235 INFO:teuthology.orchestra.run.smithi077.stdout:(12/23): python3-rgw-17.0.0-7135.gcae1ffd5.el8. 3.2 MB/s | 109 kB 00:00 2021-08-21T14:36:23.258 INFO:teuthology.orchestra.run.smithi077.stdout:(13/23): librdkafka-0.11.4-1.el8.x86_64.rpm 8.4 MB/s | 353 kB 00:00 2021-08-21T14:36:23.259 INFO:teuthology.orchestra.run.smithi077.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 34 MB/s | 44 kB 00:00 2021-08-21T14:36:23.259 INFO:teuthology.orchestra.run.smithi077.stdout:(15/23): librabbitmq-0.9.0-3.el8.x86_64.rpm 35 MB/s | 47 kB 00:00 2021-08-21T14:36:23.259 INFO:teuthology.orchestra.run.smithi077.stdout:(16/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.8 MB/s | 259 kB 00:00 2021-08-21T14:36:23.466 INFO:teuthology.orchestra.run.smithi077.stdout:(17/23): ceph-common-17.0.0-7135.gcae1ffd5.el8. 12 MB/s | 20 MB 00:01 2021-08-21T14:36:23.511 INFO:teuthology.orchestra.run.smithi077.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 876 kB/s | 181 kB 00:00 2021-08-21T14:36:23.512 INFO:teuthology.orchestra.run.smithi077.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 3.3 MB/s | 75 kB 00:00 2021-08-21T14:36:23.512 INFO:teuthology.orchestra.run.smithi077.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2021-08-21T14:36:23.578 INFO:teuthology.orchestra.run.smithi077.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 518 kB/s | 59 kB 00:00 2021-08-21T14:36:23.728 INFO:teuthology.orchestra.run.smithi077.stdout:(22/23): librbd1-17.0.0-7135.gcae1ffd5.el8.x86_ 16 MB/s | 3.9 MB 00:00 2021-08-21T14:36:23.797 INFO:teuthology.orchestra.run.smithi077.stdout:(23/23): librados2-17.0.0-7135.gcae1ffd5.el8.x8 12 MB/s | 3.4 MB 00:00 2021-08-21T14:36:23.797 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:36:23.797 INFO:teuthology.orchestra.run.smithi077.stdout:Total 25 MB/s | 48 MB 00:01 2021-08-21T14:36:23.798 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:36:23.823 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:36:23.824 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:36:24.215 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:36:24.215 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:36:25.208 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:36:25.386 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2021-08-21T14:36:25.548 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-08-21T14:36:25.919 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2021-08-21T14:36:26.108 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librados2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 3/25 2021-08-21T14:36:26.170 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 3/25 2021-08-21T14:36:26.398 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rados-2:17.0.0-7135.gcae1ffd5.el8.x86_64 4/25 2021-08-21T14:36:26.540 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 5/25 2021-08-21T14:36:26.953 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 5/25 2021-08-21T14:36:27.120 INFO:teuthology.orchestra.run.smithi077.stdout: Upgrading : librbd1-2:17.0.0-7135.gcae1ffd5.el8.x86_64 6/25 2021-08-21T14:36:27.186 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-2:17.0.0-7135.gcae1ffd5.el8.x86_64 6/25 2021-08-21T14:36:27.389 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 7/25 2021-08-21T14:36:27.584 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2021-08-21T14:36:27.946 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2021-08-21T14:36:28.129 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librgw2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 9/25 2021-08-21T14:36:28.174 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librgw2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 9/25 2021-08-21T14:36:28.400 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2021-08-21T14:36:28.537 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-argparse-2:17.0.0-7135.gcae1ffd5.el8. 11/25 2021-08-21T14:36:28.712 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cephfs-2:17.0.0-7135.gcae1ffd5.el8.x86_64 12/25 2021-08-21T14:36:28.899 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 13/25 2021-08-21T14:36:29.084 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 14/25 2021-08-21T14:36:29.257 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libradosstriper1-2:17.0.0-7135.gcae1ffd5.el8.x86_6 15/25 2021-08-21T14:36:29.322 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libradosstriper1-2:17.0.0-7135.gcae1ffd5.el8.x86_6 15/25 2021-08-21T14:36:29.552 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2021-08-21T14:36:29.822 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2021-08-21T14:36:30.141 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2021-08-21T14:36:30.142 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x8 19/25 2021-08-21T14:36:34.057 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x86_64 20/25 2021-08-21T14:36:34.257 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x86_64 20/25 2021-08-21T14:36:34.843 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x86_64 20/25 2021-08-21T14:36:35.079 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-base-2:17.0.0-7135.gcae1ffd5.el8.x86_64 21/25 2021-08-21T14:36:35.175 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-base-2:17.0.0-7135.gcae1ffd5.el8.x86_64 21/25 2021-08-21T14:36:35.299 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-selinux-2:17.0.0-7135.gcae1ffd5.el8.x86_64 22/25 2021-08-21T14:36:50.755 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-selinux-2:17.0.0-7135.gcae1ffd5.el8.x86_64 22/25 2021-08-21T14:36:50.756 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /sys 2021-08-21T14:36:50.756 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /proc 2021-08-21T14:36:50.756 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2021-08-21T14:36:50.756 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /run 2021-08-21T14:36:50.757 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /mnt 2021-08-21T14:36:50.757 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /var/tmp 2021-08-21T14:36:50.794 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /home 2021-08-21T14:36:50.794 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /tmp 2021-08-21T14:36:50.794 INFO:teuthology.orchestra.run.smithi077.stdout:skipping the directory /dev 2021-08-21T14:36:50.795 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:51.809 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-radosgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 23/25 2021-08-21T14:36:51.867 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 23/25 2021-08-21T14:36:51.868 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-08-21T14:36:52.136 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2021-08-21T14:36:52.136 INFO:teuthology.orchestra.run.smithi077.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2021-08-21T14:36:52.826 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2021-08-21T14:36:52.826 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-base-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/25 2021-08-21T14:36:53.064 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2/25 2021-08-21T14:36:53.064 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-radosgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 3/25 2021-08-21T14:36:53.064 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-selinux-2:17.0.0-7135.gcae1ffd5.el8.x86_64 4/25 2021-08-21T14:36:53.065 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 5/25 2021-08-21T14:36:53.065 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libradosstriper1-2:17.0.0-7135.gcae1ffd5.el8.x86_6 6/25 2021-08-21T14:36:53.065 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librgw2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 7/25 2021-08-21T14:36:53.161 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-argparse-2:17.0.0-7135.gcae1ffd5.el8. 8/25 2021-08-21T14:36:53.161 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x8 9/25 2021-08-21T14:36:53.161 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cephfs-2:17.0.0-7135.gcae1ffd5.el8.x86_64 10/25 2021-08-21T14:36:53.162 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rados-2:17.0.0-7135.gcae1ffd5.el8.x86_64 11/25 2021-08-21T14:36:53.285 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 12/25 2021-08-21T14:36:53.359 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 13/25 2021-08-21T14:36:53.360 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2021-08-21T14:36:53.360 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2021-08-21T14:36:53.360 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2021-08-21T14:36:53.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 17/25 2021-08-21T14:36:53.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2021-08-21T14:36:53.414 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2021-08-21T14:36:53.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2021-08-21T14:36:53.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2021-08-21T14:36:53.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 22/25 2021-08-21T14:36:53.520 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2021-08-21T14:36:53.521 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-2:17.0.0-7135.gcae1ffd5.el8.x86_64 24/25 2021-08-21T14:36:53.647 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2021-08-21T14:36:53.648 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:53.648 INFO:teuthology.orchestra.run.smithi077.stdout:Upgraded: 2021-08-21T14:36:53.648 INFO:teuthology.orchestra.run.smithi077.stdout: librados2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.648 INFO:teuthology.orchestra.run.smithi077.stdout: librbd1-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.649 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:53.649 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:36:53.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.649 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-radosgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.650 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-selinux-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.650 INFO:teuthology.orchestra.run.smithi077.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2021-08-21T14:36:53.785 INFO:teuthology.orchestra.run.smithi077.stdout: leveldb-1.22-1.el8.x86_64 2021-08-21T14:36:53.785 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.785 INFO:teuthology.orchestra.run.smithi077.stdout: liboath-2.6.2-3.el8.x86_64 2021-08-21T14:36:53.786 INFO:teuthology.orchestra.run.smithi077.stdout: librabbitmq-0.9.0-3.el8.x86_64 2021-08-21T14:36:53.786 INFO:teuthology.orchestra.run.smithi077.stdout: libradosstriper1-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.786 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-08-21T14:36:53.802 INFO:teuthology.orchestra.run.smithi077.stdout: librgw2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.803 INFO:teuthology.orchestra.run.smithi077.stdout: libunwind-1.3.1-3.el8.x86_64 2021-08-21T14:36:53.803 INFO:teuthology.orchestra.run.smithi077.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-08-21T14:36:53.803 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.819 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.873 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cephfs-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.879 INFO:teuthology.orchestra.run.smithi077.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-08-21T14:36:53.880 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rados-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.880 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.880 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:36:53.880 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:53.880 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:36:53.959 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-test 2021-08-21T14:36:54.457 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:35 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:36:55.440 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:36:55.440 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:36:55.440 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:36:55.441 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:36:55.441 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:36:55.441 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 43 M 2021-08-21T14:36:55.441 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:36:55.441 INFO:teuthology.orchestra.run.smithi077.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-08-21T14:36:55.442 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 170 k 2021-08-21T14:36:55.442 INFO:teuthology.orchestra.run.smithi077.stdout: libxslt x86_64 1.1.32-6.el8 CentOS-Base 250 k 2021-08-21T14:36:55.442 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-08-21T14:36:55.487 INFO:teuthology.orchestra.run.smithi077.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-08-21T14:36:55.488 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-08-21T14:36:55.488 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:55.488 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:36:55.488 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:36:55.511 INFO:teuthology.orchestra.run.smithi077.stdout:Install 7 Packages 2021-08-21T14:36:55.511 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:36:55.511 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 44 M 2021-08-21T14:36:55.512 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 207 M 2021-08-21T14:36:55.521 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:36:55.538 INFO:teuthology.orchestra.run.smithi077.stdout:(1/7): jq-1.5-12.el8.x86_64.rpm 4.1 MB/s | 161 kB 00:00 2021-08-21T14:36:55.551 INFO:teuthology.orchestra.run.smithi077.stdout:(2/7): oniguruma-6.8.2-2.el8.x86_64.rpm 60 MB/s | 187 kB 00:00 2021-08-21T14:36:55.551 INFO:teuthology.orchestra.run.smithi077.stdout:(3/7): socat-1.7.3.3-2.el8.x86_64.rpm 69 MB/s | 302 kB 00:00 2021-08-21T14:36:55.551 INFO:teuthology.orchestra.run.smithi077.stdout:(4/7): libxslt-1.1.32-6.el8.x86_64.rpm 66 MB/s | 250 kB 00:00 2021-08-21T14:36:55.590 INFO:teuthology.orchestra.run.smithi077.stdout:(5/7): xmlstarlet-1.6.1-11.el8.x86_64.rpm 724 kB/s | 69 kB 00:00 2021-08-21T14:36:55.626 INFO:teuthology.orchestra.run.smithi077.stdout:(6/7): libcephsqlite-17.0.0-7135.gcae1ffd5.el8. 966 kB/s | 170 kB 00:00 2021-08-21T14:36:57.479 INFO:teuthology.orchestra.run.smithi077.stdout:(7/7): ceph-test-17.0.0-7135.gcae1ffd5.el8.x86_ 21 MB/s | 43 MB 00:02 2021-08-21T14:36:57.479 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:36:57.480 INFO:teuthology.orchestra.run.smithi077.stdout:Total 22 MB/s | 44 MB 00:02 2021-08-21T14:36:57.480 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:36:57.505 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:36:57.506 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:36:57.772 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:36:57.840 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:36:58.205 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:36:58.402 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libxslt-1.1.32-6.el8.x86_64 1/7 2021-08-21T14:36:58.589 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/7 2021-08-21T14:36:58.766 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/7 2021-08-21T14:36:58.962 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-08-21T14:36:59.017 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/7 2021-08-21T14:36:59.211 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : jq-1.5-12.el8.x86_64 5/7 2021-08-21T14:36:59.372 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephsqlite-2:17.0.0-7135.gcae1ffd5.el8.x86_64 6/7 2021-08-21T14:37:03.644 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephsqlite-2:17.0.0-7135.gcae1ffd5.el8.x86_64 6/7 2021-08-21T14:37:04.067 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-test-2:17.0.0-7135.gcae1ffd5.el8.x86_64 7/7 2021-08-21T14:37:04.176 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-test-2:17.0.0-7135.gcae1ffd5.el8.x86_64 7/7 2021-08-21T14:37:04.197 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-test-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/7 2021-08-21T14:37:04.202 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephsqlite-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2/7 2021-08-21T14:37:04.202 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : jq-1.5-12.el8.x86_64 3/7 2021-08-21T14:37:04.203 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/7 2021-08-21T14:37:04.203 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 5/7 2021-08-21T14:37:04.203 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libxslt-1.1.32-6.el8.x86_64 6/7 2021-08-21T14:37:04.505 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 7/7 2021-08-21T14:37:04.556 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:04.556 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:04.556 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-test-2:17.0.0-7135.gcae1ffd5.el8.x86_64 jq-1.5-12.el8.x86_64 2021-08-21T14:37:04.556 INFO:teuthology.orchestra.run.smithi077.stdout: libcephsqlite-2:17.0.0-7135.gcae1ffd5.el8.x86_64 libxslt-1.1.32-6.el8.x86_64 2021-08-21T14:37:04.557 INFO:teuthology.orchestra.run.smithi077.stdout: oniguruma-6.8.2-2.el8.x86_64 socat-1.7.3.3-2.el8.x86_64 2021-08-21T14:37:04.557 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet-1.6.1-11.el8.x86_64 2021-08-21T14:37:04.557 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:04.557 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:04.683 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph 2021-08-21T14:37:05.198 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:06.175 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:06.175 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2021-08-21T14:37:06.175 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:37:06.176 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2021-08-21T14:37:06.176 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:37:06.176 INFO:teuthology.orchestra.run.smithi077.stdout: ceph x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 6.4 k 2021-08-21T14:37:06.176 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:37:06.177 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 2.3 M 2021-08-21T14:37:06.177 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 1.5 M 2021-08-21T14:37:06.177 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 211 k 2021-08-21T14:37:06.177 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 3.9 M 2021-08-21T14:37:06.177 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 17 M 2021-08-21T14:37:06.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-08-21T14:37:06.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-08-21T14:37:06.178 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi x86_64 1.11.5-5.el8 CentOS-Base 237 k 2021-08-21T14:37:06.257 INFO:teuthology.orchestra.run.smithi077.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-08-21T14:37:06.258 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-08-21T14:37:06.258 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-08-21T14:37:06.259 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography x86_64 3.2.1-4.el8 CentOS-Base 559 k 2021-08-21T14:37:06.259 INFO:teuthology.orchestra.run.smithi077.stdout: python3-idna noarch 2.5-5.el8 CentOS-Base 97 k 2021-08-21T14:37:06.259 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-08-21T14:37:06.260 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-08-21T14:37:06.260 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-08-21T14:37:06.261 INFO:teuthology.orchestra.run.smithi077.stdout: python3-lxml x86_64 4.2.3-2.el8 CentOS-AppStream 1.5 M 2021-08-21T14:37:06.261 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-08-21T14:37:06.261 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-08-21T14:37:06.262 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-08-21T14:37:06.262 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-08-21T14:37:06.262 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-08-21T14:37:06.271 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 CentOS-AppStream 103 k 2021-08-21T14:37:06.294 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser noarch 2.14-14.el8 CentOS-Base 109 k 2021-08-21T14:37:06.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-08-21T14:37:06.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-08-21T14:37:06.303 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-08-21T14:37:06.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-08-21T14:37:06.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-08-21T14:37:06.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-08-21T14:37:06.304 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-08-21T14:37:06.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3 noarch 1.24.2-5.el8 CentOS-Base 177 k 2021-08-21T14:37:06.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-08-21T14:37:06.305 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-08-21T14:37:06.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-08-21T14:37:06.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-08-21T14:37:06.306 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-08-21T14:37:06.415 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-08-21T14:37:06.415 INFO:teuthology.orchestra.run.smithi077.stdout:Installing weak dependencies: 2021-08-21T14:37:06.416 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-08-21T14:37:06.416 INFO:teuthology.orchestra.run.smithi077.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-08-21T14:37:06.416 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:06.449 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:37:06.449 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================= 2021-08-21T14:37:06.449 INFO:teuthology.orchestra.run.smithi077.stdout:Install 41 Packages 2021-08-21T14:37:06.449 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:06.450 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 31 M 2021-08-21T14:37:06.450 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 121 M 2021-08-21T14:37:06.450 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:37:06.464 INFO:teuthology.orchestra.run.smithi077.stdout:(1/41): ceph-17.0.0-7135.gcae1ffd5.el8.x86_64.r 60 kB/s | 6.4 kB 00:00 2021-08-21T14:37:06.464 INFO:teuthology.orchestra.run.smithi077.stdout:(2/41): ceph-mds-17.0.0-7135.gcae1ffd5.el8.x86_ 8.1 MB/s | 2.3 MB 00:00 2021-08-21T14:37:06.513 INFO:teuthology.orchestra.run.smithi077.stdout:(3/41): ceph-mgr-17.0.0-7135.gcae1ffd5.el8.x86_ 4.1 MB/s | 1.5 MB 00:00 2021-08-21T14:37:06.625 INFO:teuthology.orchestra.run.smithi077.stdout:(4/41): ceph-mgr-modules-core-17.0.0-7135.gcae1 2.6 MB/s | 211 kB 00:00 2021-08-21T14:37:06.625 INFO:teuthology.orchestra.run.smithi077.stdout:(5/41): python3-html5lib-0.999999999-6.el8.noar 6.8 MB/s | 214 kB 00:00 2021-08-21T14:37:06.643 INFO:teuthology.orchestra.run.smithi077.stdout:(6/41): python3-lxml-4.2.3-2.el8.x86_64.rpm 83 MB/s | 1.5 MB 00:00 2021-08-21T14:37:06.643 INFO:teuthology.orchestra.run.smithi077.stdout:(7/41): python3-mako-1.0.6-13.el8.noarch.rpm 56 MB/s | 157 kB 00:00 2021-08-21T14:37:06.644 INFO:teuthology.orchestra.run.smithi077.stdout:(8/41): python3-markupsafe-0.23-19.el8.x86_64.r 33 MB/s | 39 kB 00:00 2021-08-21T14:37:06.728 INFO:teuthology.orchestra.run.smithi077.stdout:(9/41): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 50 MB/s | 103 kB 00:00 2021-08-21T14:37:06.728 INFO:teuthology.orchestra.run.smithi077.stdout:(10/41): python3-pytz-2017.2-9.el8.noarch.rpm 38 MB/s | 54 kB 00:00 2021-08-21T14:37:06.729 INFO:teuthology.orchestra.run.smithi077.stdout:(11/41): python3-webencodings-0.5.1-6.el8.noarc 25 MB/s | 27 kB 00:00 2021-08-21T14:37:06.729 INFO:teuthology.orchestra.run.smithi077.stdout:(12/41): python3-werkzeug-0.12.2-4.el8.noarch.r 76 MB/s | 457 kB 00:00 2021-08-21T14:37:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:(13/41): python3-cffi-1.11.5-5.el8.x86_64.rpm 67 MB/s | 237 kB 00:00 2021-08-21T14:37:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:(14/41): python3-chardet-3.0.4-7.el8.noarch.rpm 64 MB/s | 195 kB 00:00 2021-08-21T14:37:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:(15/41): python3-cryptography-3.2.1-4.el8.x86_6 78 MB/s | 559 kB 00:00 2021-08-21T14:37:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:(16/41): python3-idna-2.5-5.el8.noarch.rpm 48 MB/s | 97 kB 00:00 2021-08-21T14:37:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:(17/41): python3-pycparser-2.14-14.el8.noarch.r 50 MB/s | 109 kB 00:00 2021-08-21T14:37:06.763 INFO:teuthology.orchestra.run.smithi077.stdout:(18/41): python3-pysocks-1.6.8-3.el8.noarch.rpm 29 MB/s | 34 kB 00:00 2021-08-21T14:37:06.763 INFO:teuthology.orchestra.run.smithi077.stdout:(19/41): python3-requests-2.20.0-2.1.el8_1.noar 54 MB/s | 123 kB 00:00 2021-08-21T14:37:06.763 INFO:teuthology.orchestra.run.smithi077.stdout:(20/41): python3-urllib3-1.24.2-5.el8.noarch.rp 61 MB/s | 177 kB 00:00 2021-08-21T14:37:06.808 INFO:teuthology.orchestra.run.smithi077.stdout:(21/41): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 389 kB/s | 44 kB 00:00 2021-08-21T14:37:06.873 INFO:teuthology.orchestra.run.smithi077.stdout:(22/41): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.3 MB/s | 185 kB 00:00 2021-08-21T14:37:06.929 INFO:teuthology.orchestra.run.smithi077.stdout:(23/41): python3-cheroot-8.5.2-1.el8.noarch.rpm 3.1 MB/s | 173 kB 00:00 2021-08-21T14:37:07.016 INFO:teuthology.orchestra.run.smithi077.stdout:(24/41): python3-cherrypy-18.4.0-1.el8.noarch.r 4.6 MB/s | 384 kB 00:00 2021-08-21T14:37:07.058 INFO:teuthology.orchestra.run.smithi077.stdout:(25/41): python3-cssselect-0.9.2-10.el8.noarch. 1.4 MB/s | 40 kB 00:00 2021-08-21T14:37:07.065 INFO:teuthology.orchestra.run.smithi077.stdout:(26/41): python3-jaraco-6.2-6.el8.noarch.rpm 402 kB/s | 11 kB 00:00 2021-08-21T14:37:07.195 INFO:teuthology.orchestra.run.smithi077.stdout:(27/41): ceph-osd-17.0.0-7135.gcae1ffd5.el8.x86 22 MB/s | 17 MB 00:00 2021-08-21T14:37:07.196 INFO:teuthology.orchestra.run.smithi077.stdout:(28/41): python3-jaraco-functools-2.0-4.el8.noa 134 kB/s | 18 kB 00:00 2021-08-21T14:37:07.260 INFO:teuthology.orchestra.run.smithi077.stdout:(29/41): python3-more-itertools-7.2.0-3.el8.noa 2.1 MB/s | 59 kB 00:00 2021-08-21T14:37:07.287 INFO:teuthology.orchestra.run.smithi077.stdout:(30/41): python3-pecan-1.3.2-9.el8.noarch.rpm 5.0 MB/s | 283 kB 00:00 2021-08-21T14:37:07.311 INFO:teuthology.orchestra.run.smithi077.stdout:(31/41): python3-portend-2.6-1.el8.noarch.rpm 563 kB/s | 16 kB 00:00 2021-08-21T14:37:07.312 INFO:teuthology.orchestra.run.smithi077.stdout:(32/41): python3-logutils-0.3.5-11.el8.noarch.r 433 kB/s | 49 kB 00:00 2021-08-21T14:37:07.335 INFO:teuthology.orchestra.run.smithi077.stdout:(33/41): python3-simplegeneric-0.8.1-17.el8.noa 660 kB/s | 19 kB 00:00 2021-08-21T14:37:07.348 INFO:teuthology.orchestra.run.smithi077.stdout:(34/41): python3-singledispatch-3.4.0.3-18.el8. 834 kB/s | 24 kB 00:00 2021-08-21T14:37:07.363 INFO:teuthology.orchestra.run.smithi077.stdout:(35/41): python3-tempora-1.14.1-5.el8.noarch.rp 1.1 MB/s | 29 kB 00:00 2021-08-21T14:37:07.363 INFO:teuthology.orchestra.run.smithi077.stdout:(36/41): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-08-21T14:37:07.441 INFO:teuthology.orchestra.run.smithi077.stdout:(37/41): python3-waitress-1.2.1-2.el8.1.noarch. 4.3 MB/s | 241 kB 00:00 2021-08-21T14:37:07.444 INFO:teuthology.orchestra.run.smithi077.stdout:(38/41): python3-webtest-2.0.33-1.el8.noarch.rp 3.0 MB/s | 85 kB 00:00 2021-08-21T14:37:07.450 INFO:teuthology.orchestra.run.smithi077.stdout:(39/41): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.9 MB/s | 251 kB 00:00 2021-08-21T14:37:07.472 INFO:teuthology.orchestra.run.smithi077.stdout:(40/41): python3-zc-lockfile-2.0-2.el8.noarch.r 847 kB/s | 23 kB 00:00 2021-08-21T14:37:07.493 INFO:teuthology.orchestra.run.smithi077.stdout:(41/41): ceph-mon-17.0.0-7135.gcae1ffd5.el8.x86 3.1 MB/s | 3.9 MB 00:01 2021-08-21T14:37:07.574 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:37:07.574 INFO:teuthology.orchestra.run.smithi077.stdout:Total 23 MB/s | 31 MB 00:01 2021-08-21T14:37:07.575 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:37:07.576 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:37:07.576 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:37:07.830 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:37:07.831 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:37:10.199 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:37:10.411 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/41 2021-08-21T14:37:10.659 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/41 2021-08-21T14:37:10.815 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-idna-2.5-5.el8.noarch 3/41 2021-08-21T14:37:11.052 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 4/41 2021-08-21T14:37:11.240 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 5/41 2021-08-21T14:37:11.429 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/41 2021-08-21T14:37:11.563 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/41 2021-08-21T14:37:11.739 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/41 2021-08-21T14:37:11.882 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/41 2021-08-21T14:37:12.031 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/41 2021-08-21T14:37:12.209 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 11/41 2021-08-21T14:37:12.443 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 12/41 2021-08-21T14:37:12.643 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-urllib3-1.24.2-5.el8.noarch 13/41 2021-08-21T14:37:12.904 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 14/41 2021-08-21T14:37:13.298 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-08-21T14:37:13.568 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cryptography-3.2.1-4.el8.x86_64 16/41 2021-08-21T14:37:13.778 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 17/41 2021-08-21T14:37:14.013 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 18/41 2021-08-21T14:37:14.165 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 19/41 2021-08-21T14:37:14.468 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 20/41 2021-08-21T14:37:14.624 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 21/41 2021-08-21T14:37:14.852 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 22/41 2021-08-21T14:37:15.051 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 23/41 2021-08-21T14:37:15.291 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 24/41 2021-08-21T14:37:15.546 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 25/41 2021-08-21T14:37:15.925 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 26/41 2021-08-21T14:37:16.195 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-lxml-4.2.3-2.el8.x86_64 27/41 2021-08-21T14:37:16.400 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 28/41 2021-08-21T14:37:16.567 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 29/41 2021-08-21T14:37:16.789 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 30/41 2021-08-21T14:37:17.031 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-portend-2.6-1.el8.noarch 31/41 2021-08-21T14:37:17.237 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 32/41 2021-08-21T14:37:17.471 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 33/41 2021-08-21T14:37:17.827 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 34/41 2021-08-21T14:37:18.192 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 35/41 2021-08-21T14:37:18.521 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-7135.gcae1ffd5.el8. 36/41 2021-08-21T14:37:18.665 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-2:17.0.0-7135.gcae1ffd5.el8.x86_64 37/41 2021-08-21T14:37:20.275 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-2:17.0.0-7135.gcae1ffd5.el8.x86_64 37/41 2021-08-21T14:37:20.561 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-osd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 38/41 2021-08-21T14:37:20.931 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-osd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 38/41 2021-08-21T14:37:21.135 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mon-2:17.0.0-7135.gcae1ffd5.el8.x86_64 39/41 2021-08-21T14:37:21.331 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mon-2:17.0.0-7135.gcae1ffd5.el8.x86_64 39/41 2021-08-21T14:37:21.533 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mds-2:17.0.0-7135.gcae1ffd5.el8.x86_64 40/41 2021-08-21T14:37:21.543 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mds-2:17.0.0-7135.gcae1ffd5.el8.x86_64 40/41 2021-08-21T14:37:21.676 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-2:17.0.0-7135.gcae1ffd5.el8.x86_64 41/41 2021-08-21T14:37:22.133 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-2:17.0.0-7135.gcae1ffd5.el8.x86_64 41/41 2021-08-21T14:37:22.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/41 2021-08-21T14:37:22.133 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mds-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2/41 2021-08-21T14:37:22.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-2:17.0.0-7135.gcae1ffd5.el8.x86_64 3/41 2021-08-21T14:37:22.134 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mon-2:17.0.0-7135.gcae1ffd5.el8.x86_64 4/41 2021-08-21T14:37:22.183 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-osd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 5/41 2021-08-21T14:37:22.184 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-7135.gcae1ffd5.el8. 6/41 2021-08-21T14:37:22.184 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/41 2021-08-21T14:37:22.193 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-lxml-4.2.3-2.el8.x86_64 8/41 2021-08-21T14:37:22.194 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/41 2021-08-21T14:37:22.195 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/41 2021-08-21T14:37:22.195 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 11/41 2021-08-21T14:37:22.195 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 12/41 2021-08-21T14:37:22.196 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 13/41 2021-08-21T14:37:22.196 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 14/41 2021-08-21T14:37:22.197 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 15/41 2021-08-21T14:37:22.197 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 16/41 2021-08-21T14:37:22.207 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cryptography-3.2.1-4.el8.x86_64 17/41 2021-08-21T14:37:22.207 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-idna-2.5-5.el8.noarch 18/41 2021-08-21T14:37:22.219 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 19/41 2021-08-21T14:37:22.220 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 20/41 2021-08-21T14:37:22.220 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 21/41 2021-08-21T14:37:22.220 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-urllib3-1.24.2-5.el8.noarch 22/41 2021-08-21T14:37:22.220 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 23/41 2021-08-21T14:37:22.221 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 24/41 2021-08-21T14:37:22.221 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 25/41 2021-08-21T14:37:22.221 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 26/41 2021-08-21T14:37:22.221 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 27/41 2021-08-21T14:37:22.222 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 28/41 2021-08-21T14:37:22.437 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 29/41 2021-08-21T14:37:22.437 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 30/41 2021-08-21T14:37:22.437 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 31/41 2021-08-21T14:37:22.438 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 32/41 2021-08-21T14:37:22.438 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-portend-2.6-1.el8.noarch 33/41 2021-08-21T14:37:22.438 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 34/41 2021-08-21T14:37:22.477 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 35/41 2021-08-21T14:37:22.477 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 36/41 2021-08-21T14:37:22.477 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 37/41 2021-08-21T14:37:22.478 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 38/41 2021-08-21T14:37:22.478 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 39/41 2021-08-21T14:37:22.478 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 40/41 2021-08-21T14:37:23.082 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 41/41 2021-08-21T14:37:23.083 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:23.083 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:23.156 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:37:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mds-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:37:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:37:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:23.157 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mon-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:37:23.168 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-osd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:37:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-08-21T14:37:23.169 INFO:teuthology.orchestra.run.smithi077.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-08-21T14:37:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cffi-1.11.5-5.el8.x86_64 2021-08-21T14:37:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-08-21T14:37:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-08-21T14:37:23.172 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-08-21T14:37:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cryptography-3.2.1-4.el8.x86_64 2021-08-21T14:37:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-08-21T14:37:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-08-21T14:37:23.173 INFO:teuthology.orchestra.run.smithi077.stdout: python3-idna-2.5-5.el8.noarch 2021-08-21T14:37:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-6.2-6.el8.noarch 2021-08-21T14:37:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-08-21T14:37:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-08-21T14:37:23.174 INFO:teuthology.orchestra.run.smithi077.stdout: python3-lxml-4.2.3-2.el8.x86_64 2021-08-21T14:37:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-mako-1.0.6-13.el8.noarch 2021-08-21T14:37:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-08-21T14:37:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-08-21T14:37:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-08-21T14:37:23.175 INFO:teuthology.orchestra.run.smithi077.stdout: python3-portend-2.6-1.el8.noarch 2021-08-21T14:37:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2021-08-21T14:37:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pycparser-2.14-14.el8.noarch 2021-08-21T14:37:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-08-21T14:37:23.176 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pytz-2017.2-9.el8.noarch 2021-08-21T14:37:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-08-21T14:37:23.177 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-08-21T14:37:23.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-08-21T14:37:23.446 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-08-21T14:37:23.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-08-21T14:37:23.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-urllib3-1.24.2-5.el8.noarch 2021-08-21T14:37:23.447 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-08-21T14:37:23.490 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-08-21T14:37:23.491 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-08-21T14:37:23.492 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-08-21T14:37:23.522 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-08-21T14:37:23.522 INFO:teuthology.orchestra.run.smithi077.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-08-21T14:37:23.523 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:23.523 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:23.523 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-base 2021-08-21T14:37:24.108 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:04 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:24.697 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-base-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:37:24.949 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:24.950 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:37:24.950 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:24.987 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install cephadm 2021-08-21T14:37:25.511 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:06 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:26.378 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:26.444 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:26.452 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:37:26.452 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:26.496 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:37:26.496 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 69 k 2021-08-21T14:37:26.496 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:26.496 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:37:26.496 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:26.523 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:37:26.531 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:26.531 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 69 k 2021-08-21T14:37:26.531 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 308 k 2021-08-21T14:37:26.531 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:37:26.533 INFO:teuthology.orchestra.run.smithi077.stdout:cephadm-17.0.0-7135.gcae1ffd5.el8.noarch.rpm 524 kB/s | 69 kB 00:00 2021-08-21T14:37:26.737 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:37:26.737 INFO:teuthology.orchestra.run.smithi077.stdout:Total 519 kB/s | 69 kB 00:00 2021-08-21T14:37:26.738 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:37:26.738 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:37:26.738 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:37:26.778 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:37:26.779 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:37:26.779 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:37:28.628 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 1/1 2021-08-21T14:37:28.881 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 1/1 2021-08-21T14:37:29.094 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 1/1 2021-08-21T14:37:29.359 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 1/1 2021-08-21T14:37:29.359 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:29.360 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:29.360 INFO:teuthology.orchestra.run.smithi077.stdout: cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:29.360 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:29.360 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:29.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-immutable-object-cache 2021-08-21T14:37:30.039 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:11 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:31.435 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:31.436 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:31.436 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2021-08-21T14:37:31.436 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:31.436 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:37:31.437 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 169 k 2021-08-21T14:37:31.437 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:31.437 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:37:31.437 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:31.437 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:37:31.438 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:31.438 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 169 k 2021-08-21T14:37:31.555 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 567 k 2021-08-21T14:37:31.555 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:37:31.585 INFO:teuthology.orchestra.run.smithi077.stdout:ceph-immutable-object-cache-17.0.0-7135.gcae1ff 877 kB/s | 169 kB 00:00 2021-08-21T14:37:31.586 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:37:31.586 INFO:teuthology.orchestra.run.smithi077.stdout:Total 871 kB/s | 169 kB 00:00 2021-08-21T14:37:31.586 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:37:31.642 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:37:31.642 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:37:31.761 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:37:31.814 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:37:31.815 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:37:32.034 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-7135.gcae1ffd5. 1/1 2021-08-21T14:37:32.367 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-7135.gcae1ffd5. 1/1 2021-08-21T14:37:32.632 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-7135.gcae1ffd5. 1/1 2021-08-21T14:37:32.633 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:32.633 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:32.633 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-immutable-object-cache-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:37:32.633 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:32.634 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:32.713 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr 2021-08-21T14:37:33.191 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:14 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:33.892 INFO:teuthology.orchestra.run.smithi077.stdout:Package ceph-mgr-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:37:34.150 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:34.153 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:37:34.154 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:34.203 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-dashboard 2021-08-21T14:37:34.699 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:15 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:35.638 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:35.638 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:35.682 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:37:35.682 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:35.682 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:37:35.769 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 1.5 M 2021-08-21T14:37:35.769 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:37:35.769 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 24 k 2021-08-21T14:37:35.770 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 8.9 k 2021-08-21T14:37:35.989 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-08-21T14:37:35.990 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-08-21T14:37:35.990 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-08-21T14:37:35.990 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:36.083 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:37:36.189 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:37:36.190 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2021-08-21T14:37:36.190 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:36.190 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 1.8 M 2021-08-21T14:37:36.346 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 7.4 M 2021-08-21T14:37:36.347 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:37:36.348 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): ceph-prometheus-alerts-17.0.0-7135.gcae1 88 kB/s | 8.9 kB 00:00 2021-08-21T14:37:36.349 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): ceph-grafana-dashboards-17.0.0-7135.gcae 213 kB/s | 24 kB 00:00 2021-08-21T14:37:36.349 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-08-21T14:37:36.349 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 413 kB/s | 34 kB 00:00 2021-08-21T14:37:36.349 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2021-08-21T14:37:36.350 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): ceph-mgr-dashboard-17.0.0-7135.gcae1ffd5 4.5 MB/s | 1.5 MB 00:00 2021-08-21T14:37:36.350 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:37:36.350 INFO:teuthology.orchestra.run.smithi077.stdout:Total 5.3 MB/s | 1.8 MB 00:00 2021-08-21T14:37:36.391 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:37:36.392 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:37:36.392 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:37:36.392 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:37:36.393 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:37:37.378 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:37:37.599 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-08-21T14:37:37.841 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-08-21T14:37:38.056 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-08-21T14:37:38.263 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-7135.gcae1ffd5.el8.n 4/6 2021-08-21T14:37:38.566 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-7135.gcae1ffd5.el8. 5/6 2021-08-21T14:37:38.776 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-7135.gcae1ffd5.el8.noarc 6/6 2021-08-21T14:37:38.991 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-7135.gcae1ffd5.el8.noarc 6/6 2021-08-21T14:37:39.024 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-7135.gcae1ffd5.el8. 1/6 2021-08-21T14:37:39.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-7135.gcae1ffd5.el8.noarc 2/6 2021-08-21T14:37:39.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-7135.gcae1ffd5.el8.n 3/6 2021-08-21T14:37:39.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-08-21T14:37:39.025 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-08-21T14:37:39.335 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-08-21T14:37:39.336 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:39.336 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:39.336 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-grafana-dashboards-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:39.336 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:39.337 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-prometheus-alerts-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:39.337 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-08-21T14:37:39.337 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-08-21T14:37:39.412 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes-2.4.1-12.el8.noarch 2021-08-21T14:37:39.413 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:39.413 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:39.443 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-diskprediction-local 2021-08-21T14:37:39.902 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:20 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:40.879 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:40.880 INFO:teuthology.orchestra.run.smithi077.stdout:===================================================================================================== 2021-08-21T14:37:40.880 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:37:40.880 INFO:teuthology.orchestra.run.smithi077.stdout:===================================================================================================== 2021-08-21T14:37:40.881 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:37:40.881 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 7.4 M 2021-08-21T14:37:40.881 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:37:40.881 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran x86_64 8.4.1-1.el8 CentOS-Base 639 k 2021-08-21T14:37:40.881 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath x86_64 8.4.1-1.el8 CentOS-Base 169 k 2021-08-21T14:37:40.882 INFO:teuthology.orchestra.run.smithi077.stdout: openblas x86_64 0.3.12-1.el8 CentOS-AppStream 4.6 M 2021-08-21T14:37:40.882 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads x86_64 0.3.12-1.el8 CentOS-AppStream 4.7 M 2021-08-21T14:37:40.882 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-08-21T14:37:40.930 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-08-21T14:37:40.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy x86_64 1.0.0-20.module_el8.4.0+790+083e3d81 CentOS-AppStream 15 M 2021-08-21T14:37:40.931 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:40.931 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:37:40.963 INFO:teuthology.orchestra.run.smithi077.stdout:===================================================================================================== 2021-08-21T14:37:40.963 INFO:teuthology.orchestra.run.smithi077.stdout:Install 8 Packages 2021-08-21T14:37:40.964 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:40.964 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 36 M 2021-08-21T14:37:40.964 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 197 M 2021-08-21T14:37:41.012 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:37:41.013 INFO:teuthology.orchestra.run.smithi077.stdout:(1/8): openblas-0.3.12-1.el8.x86_64.rpm 39 MB/s | 4.6 MB 00:00 2021-08-21T14:37:41.013 INFO:teuthology.orchestra.run.smithi077.stdout:(2/8): openblas-threads-0.3.12-1.el8.x86_64.rpm 31 MB/s | 4.7 MB 00:00 2021-08-21T14:37:41.013 INFO:teuthology.orchestra.run.smithi077.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 29 MB/s | 225 kB 00:00 2021-08-21T14:37:41.030 INFO:teuthology.orchestra.run.smithi077.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 47 MB/s | 3.7 MB 00:00 2021-08-21T14:37:41.083 INFO:teuthology.orchestra.run.smithi077.stdout:(5/8): libgfortran-8.4.1-1.el8.x86_64.rpm 38 MB/s | 639 kB 00:00 2021-08-21T14:37:41.083 INFO:teuthology.orchestra.run.smithi077.stdout:(6/8): libquadmath-8.4.1-1.el8.x86_64.rpm 29 MB/s | 169 kB 00:00 2021-08-21T14:37:41.301 INFO:teuthology.orchestra.run.smithi077.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.4.0+79 54 MB/s | 15 MB 00:00 2021-08-21T14:37:42.240 INFO:teuthology.orchestra.run.smithi077.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-713 5.2 MB/s | 7.4 MB 00:01 2021-08-21T14:37:42.515 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:37:42.515 INFO:teuthology.orchestra.run.smithi077.stdout:Total 25 MB/s | 36 MB 00:01 2021-08-21T14:37:42.515 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:37:42.516 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:37:42.535 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:37:42.656 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:37:42.656 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:37:43.190 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:37:43.303 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 1/8 2021-08-21T14:37:43.443 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 1/8 2021-08-21T14:37:43.612 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 2/8 2021-08-21T14:37:44.160 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 2/8 2021-08-21T14:37:44.349 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-0.3.12-1.el8.x86_64 3/8 2021-08-21T14:37:44.873 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 3/8 2021-08-21T14:37:45.013 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-08-21T14:37:45.730 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 4/8 2021-08-21T14:37:46.077 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-08-21T14:37:46.370 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-08-21T14:37:48.379 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-08-21T14:37:49.560 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 7/8 2021-08-21T14:37:49.753 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-7135.gcae1ffd 8/8 2021-08-21T14:37:50.016 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-7135.gcae1ffd 8/8 2021-08-21T14:37:50.016 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-7135.gcae1ffd 1/8 2021-08-21T14:37:50.016 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 2/8 2021-08-21T14:37:50.016 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 3/8 2021-08-21T14:37:50.017 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-08-21T14:37:50.017 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-08-21T14:37:50.017 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x 6/8 2021-08-21T14:37:50.017 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 7/8 2021-08-21T14:37:50.326 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 8/8 2021-08-21T14:37:50.342 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:50.342 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:50.343 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-diskprediction-local-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:50.391 INFO:teuthology.orchestra.run.smithi077.stdout: libgfortran-8.4.1-1.el8.x86_64 2021-08-21T14:37:50.391 INFO:teuthology.orchestra.run.smithi077.stdout: libquadmath-8.4.1-1.el8.x86_64 2021-08-21T14:37:50.392 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-0.3.12-1.el8.x86_64 2021-08-21T14:37:50.392 INFO:teuthology.orchestra.run.smithi077.stdout: openblas-threads-0.3.12-1.el8.x86_64 2021-08-21T14:37:50.392 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-08-21T14:37:50.392 INFO:teuthology.orchestra.run.smithi077.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-08-21T14:37:50.392 INFO:teuthology.orchestra.run.smithi077.stdout: python3-scipy-1.0.0-20.module_el8.4.0+790+083e3d81.x86_64 2021-08-21T14:37:50.393 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:50.393 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:50.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-rook 2021-08-21T14:37:50.904 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:37:51.842 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:37:52.113 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================== 2021-08-21T14:37:52.113 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:37:52.113 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================== 2021-08-21T14:37:52.114 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:37:52.114 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 43 k 2021-08-21T14:37:52.114 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:37:52.265 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-08-21T14:37:52.265 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-08-21T14:37:52.384 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-08-21T14:37:52.384 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-08-21T14:37:52.384 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-08-21T14:37:52.385 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-08-21T14:37:52.579 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-08-21T14:37:52.580 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-08-21T14:37:52.580 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-08-21T14:37:52.580 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-08-21T14:37:52.580 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-08-21T14:37:52.581 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-08-21T14:37:52.581 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:52.581 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:37:52.581 INFO:teuthology.orchestra.run.smithi077.stdout:======================================================================================== 2021-08-21T14:37:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:Install 13 Packages 2021-08-21T14:37:52.888 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:52.888 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 1.9 M 2021-08-21T14:37:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 25 M 2021-08-21T14:37:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:37:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 846 kB/s | 27 kB 00:00 2021-08-21T14:37:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 40 MB/s | 126 kB 00:00 2021-08-21T14:37:52.889 INFO:teuthology.orchestra.run.smithi077.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 570 kB/s | 20 kB 00:00 2021-08-21T14:37:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 35 MB/s | 110 kB 00:00 2021-08-21T14:37:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 33 MB/s | 155 kB 00:00 2021-08-21T14:37:52.890 INFO:teuthology.orchestra.run.smithi077.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 19 MB/s | 43 kB 00:00 2021-08-21T14:37:53.572 INFO:teuthology.orchestra.run.smithi077.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 422 kB/s | 33 kB 00:00 2021-08-21T14:37:53.572 INFO:teuthology.orchestra.run.smithi077.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 208 kB/s | 16 kB 00:00 2021-08-21T14:37:53.573 INFO:teuthology.orchestra.run.smithi077.stdout:(9/13): ceph-mgr-rook-17.0.0-7135.gcae1ffd5.el8 292 kB/s | 43 kB 00:00 2021-08-21T14:37:54.048 INFO:teuthology.orchestra.run.smithi077.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.9 MB/s | 81 kB 00:00 2021-08-21T14:37:54.048 INFO:teuthology.orchestra.run.smithi077.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.7 MB/s | 61 kB 00:00 2021-08-21T14:37:54.048 INFO:teuthology.orchestra.run.smithi077.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 755 kB/s | 73 kB 00:00 2021-08-21T14:37:54.049 INFO:teuthology.orchestra.run.smithi077.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.8 MB/s | 1.2 MB 00:00 2021-08-21T14:37:54.075 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:37:54.263 INFO:teuthology.orchestra.run.smithi077.stdout:Total 6.6 MB/s | 1.9 MB 00:00 2021-08-21T14:37:54.330 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:37:54.331 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:37:54.331 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:37:54.331 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:37:54.331 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:37:54.332 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:37:54.332 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-08-21T14:37:54.656 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-08-21T14:37:54.657 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-08-21T14:37:54.657 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-08-21T14:37:54.732 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-08-21T14:37:55.030 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-08-21T14:37:55.278 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-08-21T14:37:55.509 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-08-21T14:37:56.648 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-08-21T14:37:56.981 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-08-21T14:37:57.088 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-08-21T14:37:57.298 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-08-21T14:37:57.460 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-rook-2:17.0.0-7135.gcae1ffd5.el8.noarch 13/13 2021-08-21T14:37:57.748 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-7135.gcae1ffd5.el8.noarch 13/13 2021-08-21T14:37:57.748 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-rook-2:17.0.0-7135.gcae1ffd5.el8.noarch 1/13 2021-08-21T14:37:57.749 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-08-21T14:37:57.749 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-08-21T14:37:57.749 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-08-21T14:37:57.785 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-08-21T14:37:58.186 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-08-21T14:37:58.191 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-08-21T14:37:58.191 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-08-21T14:37:58.191 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-08-21T14:37:58.192 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-08-21T14:37:58.192 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-08-21T14:37:58.192 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-08-21T14:37:58.290 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-08-21T14:37:58.290 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:58.608 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:37:58.609 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-rook-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:37:58.825 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-08-21T14:37:58.846 INFO:teuthology.orchestra.run.smithi077.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-08-21T14:37:59.044 INFO:teuthology.orchestra.run.smithi077.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-08-21T14:37:59.108 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-08-21T14:37:59.109 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-08-21T14:37:59.109 INFO:teuthology.orchestra.run.smithi077.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-08-21T14:37:59.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-08-21T14:37:59.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-08-21T14:37:59.419 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-08-21T14:37:59.420 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-08-21T14:37:59.511 INFO:teuthology.orchestra.run.smithi077.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-08-21T14:37:59.512 INFO:teuthology.orchestra.run.smithi077.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-08-21T14:37:59.512 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:37:59.512 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:37:59.513 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-mgr-cephadm 2021-08-21T14:38:00.100 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:01.089 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:01.090 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:01.090 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:38:01.205 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:01.205 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:01.206 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm noarch 2:17.0.0-7135.gcae1ffd5.el8 ceph-noarch 91 k 2021-08-21T14:38:01.289 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:38:01.289 INFO:teuthology.orchestra.run.smithi077.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-08-21T14:38:01.289 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-08-21T14:38:01.290 INFO:teuthology.orchestra.run.smithi077.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-08-21T14:38:01.290 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-08-21T14:38:01.290 INFO:teuthology.orchestra.run.smithi077.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-08-21T14:38:01.302 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:01.303 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:01.313 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:01.321 INFO:teuthology.orchestra.run.smithi077.stdout:Install 6 Packages 2021-08-21T14:38:01.321 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:01.321 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 5.8 M 2021-08-21T14:38:01.321 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 24 M 2021-08-21T14:38:01.322 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:01.361 INFO:teuthology.orchestra.run.smithi077.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 12 MB/s | 538 kB 00:00 2021-08-21T14:38:01.362 INFO:teuthology.orchestra.run.smithi077.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 50 MB/s | 4.8 MB 00:00 2021-08-21T14:38:01.362 INFO:teuthology.orchestra.run.smithi077.stdout:(3/6): ceph-mgr-cephadm-17.0.0-7135.gcae1ffd5.e 695 kB/s | 91 kB 00:00 2021-08-21T14:38:01.362 INFO:teuthology.orchestra.run.smithi077.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 240 kB/s | 21 kB 00:00 2021-08-21T14:38:01.366 INFO:teuthology.orchestra.run.smithi077.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.6 MB/s | 355 kB 00:00 2021-08-21T14:38:01.366 INFO:teuthology.orchestra.run.smithi077.stdout:(6/6): python3-remoto-1.1.4-4.el8.noarch.rpm 421 kB/s | 48 kB 00:00 2021-08-21T14:38:01.366 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:01.366 INFO:teuthology.orchestra.run.smithi077.stdout:Total 23 MB/s | 5.8 MB 00:00 2021-08-21T14:38:01.367 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:01.367 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:01.367 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:01.442 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:01.773 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:01.835 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:01.957 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-08-21T14:38:02.172 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-08-21T14:38:03.172 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-08-21T14:38:03.527 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-08-21T14:38:04.324 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-08-21T14:38:04.573 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 6/6 2021-08-21T14:38:04.791 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 6/6 2021-08-21T14:38:04.792 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 1/6 2021-08-21T14:38:04.792 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-08-21T14:38:04.792 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-08-21T14:38:04.793 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-08-21T14:38:04.793 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-08-21T14:38:05.035 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-08-21T14:38:05.035 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:05.036 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:05.036 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-cephadm-2:17.0.0-7135.gcae1ffd5.el8.noarch 2021-08-21T14:38:05.036 INFO:teuthology.orchestra.run.smithi077.stdout: python3-apipkg-1.5-6.el8.noarch 2021-08-21T14:38:05.036 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel-2.5.1-5.el8.noarch 2021-08-21T14:38:05.037 INFO:teuthology.orchestra.run.smithi077.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-08-21T14:38:05.037 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-08-21T14:38:05.037 INFO:teuthology.orchestra.run.smithi077.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-08-21T14:38:05.037 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:05.037 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:05.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install ceph-fuse 2021-08-21T14:38:05.774 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:06.686 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:06.696 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:06.696 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repository Size 2021-08-21T14:38:06.697 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:06.731 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:06.732 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 868 k 2021-08-21T14:38:06.732 INFO:teuthology.orchestra.run.smithi077.stdout:Installing dependencies: 2021-08-21T14:38:06.732 INFO:teuthology.orchestra.run.smithi077.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-08-21T14:38:06.732 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:06.733 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:06.761 INFO:teuthology.orchestra.run.smithi077.stdout:Install 2 Packages 2021-08-21T14:38:06.762 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 951 k 2021-08-21T14:38:06.762 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 3.1 M 2021-08-21T14:38:06.777 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:06.957 INFO:teuthology.orchestra.run.smithi077.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-08-21T14:38:07.852 INFO:teuthology.orchestra.run.smithi077.stdout:(2/2): ceph-fuse-17.0.0-7135.gcae1ffd5.el8.x86_ 763 kB/s | 868 kB 00:01 2021-08-21T14:38:07.852 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:07.853 INFO:teuthology.orchestra.run.smithi077.stdout:Total 833 kB/s | 951 kB 00:01 2021-08-21T14:38:07.853 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:07.853 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:07.853 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:07.930 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:07.967 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:08.092 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:08.359 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-08-21T14:38:08.635 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : ceph-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2/2 2021-08-21T14:38:08.899 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: ceph-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2/2 2021-08-21T14:38:08.899 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : ceph-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/2 2021-08-21T14:38:09.206 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-08-21T14:38:09.207 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:09.207 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:09.207 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-08-21T14:38:09.207 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:09.208 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:09.592 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados-devel 2021-08-21T14:38:10.135 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:51 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:11.147 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:11.148 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:11.148 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2021-08-21T14:38:11.148 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:11.149 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:11.149 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 125 k 2021-08-21T14:38:11.149 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:11.149 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:11.201 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:11.245 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:38:11.245 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:11.246 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 125 k 2021-08-21T14:38:11.246 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 432 k 2021-08-21T14:38:11.246 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:11.487 INFO:teuthology.orchestra.run.smithi077.stdout:librados-devel-17.0.0-7135.gcae1ffd5.el8.x86_64 321 kB/s | 125 kB 00:00 2021-08-21T14:38:11.487 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:11.487 INFO:teuthology.orchestra.run.smithi077.stdout:Total 320 kB/s | 125 kB 00:00 2021-08-21T14:38:11.488 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:11.488 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:11.488 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:11.610 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:11.644 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:11.691 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:11.839 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : librados-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:12.133 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: librados-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:12.379 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : librados-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:12.380 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: librados-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:38:12.380 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:12.381 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:12.571 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs2 2021-08-21T14:38:13.130 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:54 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:13.763 INFO:teuthology.orchestra.run.smithi077.stdout:Package libcephfs2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:14.104 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:14.105 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:14.105 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:14.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install libcephfs-devel 2021-08-21T14:38:14.677 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:55 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:15.828 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:15.828 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:15.828 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2021-08-21T14:38:15.829 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:15.829 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 24 k 2021-08-21T14:38:15.829 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:15.829 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:15.830 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:15.830 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:38:15.830 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:16.032 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 24 k 2021-08-21T14:38:16.045 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 102 k 2021-08-21T14:38:16.046 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:16.067 INFO:teuthology.orchestra.run.smithi077.stdout:libcephfs-devel-17.0.0-7135.gcae1ffd5.el8.x86_6 189 kB/s | 24 kB 00:00 2021-08-21T14:38:16.067 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:16.068 INFO:teuthology.orchestra.run.smithi077.stdout:Total 187 kB/s | 24 kB 00:00 2021-08-21T14:38:16.068 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:16.068 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:16.068 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:16.069 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:16.069 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:16.069 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:16.172 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : libcephfs-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:16.387 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:16.757 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : libcephfs-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:16.758 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:16.758 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:16.963 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-devel-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:38:16.963 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:16.964 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:17.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librados2 2021-08-21T14:38:17.757 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:01:58 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:18.497 INFO:teuthology.orchestra.run.smithi077.stdout:Package librados2-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:18.727 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:18.728 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:18.728 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:18.783 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install librbd1 2021-08-21T14:38:19.359 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:00 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:20.145 INFO:teuthology.orchestra.run.smithi077.stdout:Package librbd1-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:20.258 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:20.867 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:20.867 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:20.868 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rados 2021-08-21T14:38:21.449 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:02 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:22.116 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rados-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:22.303 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:22.307 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:22.307 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:22.394 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rgw 2021-08-21T14:38:22.962 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:03 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:23.782 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rgw-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:23.951 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:23.952 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:24.017 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-cephfs 2021-08-21T14:38:25.002 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:05 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:25.588 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-cephfs-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:25.820 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:25.833 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:25.834 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:25.978 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install python3-rbd 2021-08-21T14:38:26.915 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:07 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:27.551 INFO:teuthology.orchestra.run.smithi077.stdout:Package python3-rbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 is already installed. 2021-08-21T14:38:27.851 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:27.953 INFO:teuthology.orchestra.run.smithi077.stdout:Nothing to do. 2021-08-21T14:38:27.954 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:28.000 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-fuse 2021-08-21T14:38:28.823 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:09 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:29.512 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:29.512 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:29.512 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2021-08-21T14:38:29.512 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:29.513 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:29.513 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 93 k 2021-08-21T14:38:29.513 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:29.513 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:29.514 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:29.608 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:38:29.608 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:29.608 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 93 k 2021-08-21T14:38:29.608 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 281 k 2021-08-21T14:38:29.609 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:29.872 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-fuse-17.0.0-7135.gcae1ffd5.el8.x86_64.rpm 232 kB/s | 93 kB 00:00 2021-08-21T14:38:29.873 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:29.873 INFO:teuthology.orchestra.run.smithi077.stdout:Total 231 kB/s | 93 kB 00:00 2021-08-21T14:38:29.873 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:29.874 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:29.874 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:29.943 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:29.943 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:30.392 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:30.503 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:30.858 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:30.987 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:31.023 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:31.024 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-fuse-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:38:31.024 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:31.024 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:31.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-mirror 2021-08-21T14:38:32.612 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:13 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:33.536 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:33.536 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:33.536 INFO:teuthology.orchestra.run.smithi077.stdout: Package Arch Version Repo Size 2021-08-21T14:38:33.537 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:33.537 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:33.537 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 3.8 M 2021-08-21T14:38:33.537 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:33.538 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:33.564 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:33.565 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:38:33.571 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:33.571 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 3.8 M 2021-08-21T14:38:33.571 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 19 M 2021-08-21T14:38:33.572 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:35.243 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-mirror-17.0.0-7135.gcae1ffd5.el8.x86_64.rpm 2.2 MB/s | 3.8 MB 00:01 2021-08-21T14:38:35.243 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:35.244 INFO:teuthology.orchestra.run.smithi077.stdout:Total 2.2 MB/s | 3.8 MB 00:01 2021-08-21T14:38:35.244 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:35.307 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:35.307 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:35.308 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:35.308 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:35.803 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:35.993 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-mirror-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:36.417 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-mirror-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:36.634 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-mirror-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:36.634 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:36.707 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:36.707 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-mirror-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:38:36.707 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:36.708 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:36.796 DEBUG:teuthology.orchestra.run.smithi077:> sudo yum -y install rbd-nbd 2021-08-21T14:38:37.264 INFO:teuthology.orchestra.run.smithi077.stdout:Last metadata expiration check: 0:02:18 ago on Sat 21 Aug 2021 02:36:19 PM UTC. 2021-08-21T14:38:38.258 INFO:teuthology.orchestra.run.smithi077.stdout:Dependencies resolved. 2021-08-21T14:38:38.268 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:38.268 INFO:teuthology.orchestra.run.smithi077.stdout: Package Architecture Version Repository Size 2021-08-21T14:38:38.269 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:38.271 INFO:teuthology.orchestra.run.smithi077.stdout:Installing: 2021-08-21T14:38:38.271 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd x86_64 2:17.0.0-7135.gcae1ffd5.el8 ceph 172 k 2021-08-21T14:38:38.272 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:38.272 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction Summary 2021-08-21T14:38:38.272 INFO:teuthology.orchestra.run.smithi077.stdout:================================================================================ 2021-08-21T14:38:38.272 INFO:teuthology.orchestra.run.smithi077.stdout:Install 1 Package 2021-08-21T14:38:38.272 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:38.273 INFO:teuthology.orchestra.run.smithi077.stdout:Total download size: 172 k 2021-08-21T14:38:38.273 INFO:teuthology.orchestra.run.smithi077.stdout:Installed size: 563 k 2021-08-21T14:38:38.273 INFO:teuthology.orchestra.run.smithi077.stdout:Downloading Packages: 2021-08-21T14:38:38.433 INFO:teuthology.orchestra.run.smithi077.stdout:rbd-nbd-17.0.0-7135.gcae1ffd5.el8.x86_64.rpm 996 kB/s | 172 kB 00:00 2021-08-21T14:38:38.434 INFO:teuthology.orchestra.run.smithi077.stdout:-------------------------------------------------------------------------------- 2021-08-21T14:38:38.434 INFO:teuthology.orchestra.run.smithi077.stdout:Total 987 kB/s | 172 kB 00:00 2021-08-21T14:38:38.443 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction check 2021-08-21T14:38:38.445 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction check succeeded. 2021-08-21T14:38:38.445 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction test 2021-08-21T14:38:38.512 INFO:teuthology.orchestra.run.smithi077.stdout:Transaction test succeeded. 2021-08-21T14:38:38.513 INFO:teuthology.orchestra.run.smithi077.stdout:Running transaction 2021-08-21T14:38:38.660 INFO:teuthology.orchestra.run.smithi077.stdout: Preparing : 1/1 2021-08-21T14:38:38.818 INFO:teuthology.orchestra.run.smithi077.stdout: Installing : rbd-nbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:39.078 INFO:teuthology.orchestra.run.smithi077.stdout: Running scriptlet: rbd-nbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:39.370 INFO:teuthology.orchestra.run.smithi077.stdout: Verifying : rbd-nbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 1/1 2021-08-21T14:38:39.382 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:39.382 INFO:teuthology.orchestra.run.smithi077.stdout:Installed: 2021-08-21T14:38:39.382 INFO:teuthology.orchestra.run.smithi077.stdout: rbd-nbd-2:17.0.0-7135.gcae1ffd5.el8.x86_64 2021-08-21T14:38:39.382 INFO:teuthology.orchestra.run.smithi077.stdout: 2021-08-21T14:38:39.383 INFO:teuthology.orchestra.run.smithi077.stdout:Complete! 2021-08-21T14:38:39.547 DEBUG:teuthology.parallel:result is None 2021-08-21T14:38:39.548 DEBUG:teuthology.orchestra.run.smithi077:> sudo lsb_release -is 2021-08-21T14:38:39.735 INFO:teuthology.orchestra.run.smithi077.stdout:CentOS 2021-08-21T14:38:39.748 DEBUG:teuthology.misc:System to be installed: CentOS 2021-08-21T14:38:39.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:38:40.939 DEBUG:teuthology.orchestra.run.smithi077:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-08-21T14:38:41.000 INFO:teuthology.orchestra.run.smithi077.stdout:17.0.0-7135.gcae1ffd5.el8 2021-08-21T14:38:41.000 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-7135.gcae1ffd5.el8 2021-08-21T14:38:41.001 INFO:teuthology.task.install:The correct ceph version 17.0.0-7135.gcae1ffd5 is installed. 2021-08-21T14:38:41.001 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-08-21T14:38:41.002 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-08-21T14:38:41.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-08-21T14:38:41.112 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-08-21T14:38:41.113 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-08-21T14:38:41.113 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2021-08-21T14:38:41.276 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-08-21T14:38:41.396 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-08-21T14:38:41.396 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-08-21T14:38:41.397 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2021-08-21T14:38:41.695 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-08-21T14:38:42.222 INFO:teuthology.run_tasks:Running task workunit... 2021-08-21T14:38:43.088 INFO:tasks.workunit:Pulling workunits from ref cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:38:43.089 INFO:tasks.workunit:Making a separate scratch dir for every client... 2021-08-21T14:38:43.090 INFO:tasks.workunit:timeout=3h 2021-08-21T14:38:43.090 INFO:tasks.workunit:cleanup=True 2021-08-21T14:38:43.091 DEBUG:teuthology.orchestra.run.smithi077:> stat -- /home/ubuntu/cephtest/mnt.0 2021-08-21T14:38:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-08-21T14:38:43.241 INFO:teuthology.orchestra.run.smithi077.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2021-08-21T14:38:43.242 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -- /home/ubuntu/cephtest/mnt.0 2021-08-21T14:38:43.829 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2021-08-21T14:38:43.829 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2021-08-21T14:38:44.106 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout cae1ffd55a16af4627c36508fde05bf24312fad6 2021-08-21T14:38:44.221 INFO:tasks.workunit.client.0.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2021-08-21T14:39:18.974 INFO:tasks.workunit.client.0.smithi077.stderr:Note: switching to 'cae1ffd55a16af4627c36508fde05bf24312fad6'. 2021-08-21T14:39:19.057 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.057 INFO:tasks.workunit.client.0.smithi077.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2021-08-21T14:39:19.057 INFO:tasks.workunit.client.0.smithi077.stderr:changes and commit them, and you can discard any commits you make in this 2021-08-21T14:39:19.057 INFO:tasks.workunit.client.0.smithi077.stderr:state without impacting any branches by switching back to a branch. 2021-08-21T14:39:19.058 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.058 INFO:tasks.workunit.client.0.smithi077.stderr:If you want to create a new branch to retain commits you create, you may 2021-08-21T14:39:19.058 INFO:tasks.workunit.client.0.smithi077.stderr:do so (now or later) by using -c with the switch command. Example: 2021-08-21T14:39:19.058 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.059 INFO:tasks.workunit.client.0.smithi077.stderr: git switch -c 2021-08-21T14:39:19.059 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.068 INFO:tasks.workunit.client.0.smithi077.stderr:Or undo this operation with: 2021-08-21T14:39:19.068 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.068 INFO:tasks.workunit.client.0.smithi077.stderr: git switch - 2021-08-21T14:39:19.069 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.069 INFO:tasks.workunit.client.0.smithi077.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2021-08-21T14:39:19.069 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:39:19.069 INFO:tasks.workunit.client.0.smithi077.stderr:HEAD is now at cae1ffd55a1 Merge pull request #42813 from wjwithagen/wjw-fix-hexdump-output 2021-08-21T14:39:19.070 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2021-08-21T14:39:19.125 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2021-08-21T14:39:19.125 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2021-08-21T14:39:19.385 INFO:tasks.workunit:Running workunits matching osd-backfill on client.0... 2021-08-21T14:39:19.385 INFO:tasks.workunit:Running workunit osd-backfill/osd-backfill-prio.sh... 2021-08-21T14:39:19.658 DEBUG:teuthology.orchestra.run.smithi077:workunit test osd-backfill/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cae1ffd55a16af4627c36508fde05bf24312fad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh 2021-08-21T14:39:19.702 INFO:tasks.workunit.client.0.smithi077.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-08-21T14:39:21.080 INFO:tasks.workunit.client.0.smithi077.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-08-21T14:39:21.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-08-21T14:39:21.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-08-21T14:39:21.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: main: export PYTHONWARNINGS=ignore 2021-08-21T14:39:21.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: main: PYTHONWARNINGS=ignore 2021-08-21T14:39:21.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: main: export CEPH_CONF=/dev/null 2021-08-21T14:39:21.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: main: CEPH_CONF=/dev/null 2021-08-21T14:39:21.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: main: unset CEPH_ARGS 2021-08-21T14:39:21.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: main: local code 2021-08-21T14:39:21.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: main: run td/osd-backfill-prio 2021-08-21T14:39:21.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2021-08-21T14:39:21.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:22: run: shift 2021-08-21T14:39:21.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2021-08-21T14:39:21.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2021-08-21T14:39:21.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2021-08-21T14:39:21.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:27: run: uuidgen 2021-08-21T14:39:21.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none ' 2021-08-21T14:39:21.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:39:21.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2021-08-21T14:39:21.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:32: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2021-08-21T14:39:21.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:33: run: export objects=50 2021-08-21T14:39:21.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:33: run: objects=50 2021-08-21T14:39:21.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:34: run: export poolprefix=test 2021-08-21T14:39:21.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:34: run: poolprefix=test 2021-08-21T14:39:21.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:35: run: export FORCE_PRIO=254 2021-08-21T14:39:21.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:35: run: FORCE_PRIO=254 2021-08-21T14:39:21.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:36: run: export DEGRADED_PRIO=150 2021-08-21T14:39:21.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:36: run: DEGRADED_PRIO=150 2021-08-21T14:39:21.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:37: run: export NORMAL_PRIO=110 2021-08-21T14:39:21.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:37: run: NORMAL_PRIO=110 2021-08-21T14:39:21.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: set 2021-08-21T14:39:21.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-08-21T14:39:21.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: local 'funcs=TEST_backfill_pool_priority 2021-08-21T14:39:21.812 INFO:tasks.workunit.client.0.smithi077.stderr:TEST_backfill_priority' 2021-08-21T14:39:21.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:40: run: for func in $funcs 2021-08-21T14:39:21.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:41: run: setup td/osd-backfill-prio 2021-08-21T14:39:21.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2021-08-21T14:39:21.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2021-08-21T14:39:21.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-08-21T14:39:21.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:39:21.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-08-21T14:39:21.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:39:21.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:39:21.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:39:22.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:39:22.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:39:22.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:39:22.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:39:22.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:39:22.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:39:22.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:39:22.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:39:22.102 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:39:22.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:39:22.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:39:22.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:39:22.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:39:22.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:39:22.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:39:22.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-08-21T14:39:22.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:39:22.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:22.636 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:22.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26783 2021-08-21T14:39:22.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:39:22.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:39:22.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2021-08-21T14:39:22.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-08-21T14:39:22.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:22.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:22.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26783 2021-08-21T14:39:22.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-08-21T14:39:22.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-08-21T14:39:22.919 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-08-21T14:39:22.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-08-21T14:39:22.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2021-08-21T14:39:23.021 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: TEST_backfill_pool_priority td/osd-backfill-prio 2021-08-21T14:39:23.021 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:329: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2021-08-21T14:39:23.021 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: local pools=3 2021-08-21T14:39:23.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: local OSDS=2 2021-08-21T14:39:23.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:333: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2021-08-21T14:39:23.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2021-08-21T14:39:23.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-08-21T14:39:23.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-08-21T14:39:23.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-08-21T14:39:23.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2021-08-21T14:39:23.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2021-08-21T14:39:23.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-08-21T14:39:23.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:39:23.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:39:23.586 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:39:23.636 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:23.636 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:23.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:39:23.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-08-21T14:39:23.674 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-08-21T14:39:23.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-08-21T14:39:23.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:39:23.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:39:23.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=fsid 2021-08-21T14:39:23.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .fsid 2021-08-21T14:39:24.026 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:39:24.027 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:39:24.027 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:39:24.027 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:39:24.027 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:24.028 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:24.028 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-mon.a.asok 2021-08-21T14:39:24.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:39:24.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.26783/ceph-mon.a.asok config get fsid 2021-08-21T14:39:24.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-08-21T14:39:24.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:39:24.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:39:24.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=mon_host 2021-08-21T14:39:24.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .mon_host 2021-08-21T14:39:24.030 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:39:24.030 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:39:24.030 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:39:24.107 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:39:24.332 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:24.332 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:24.332 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-mon.a.asok 2021-08-21T14:39:24.333 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:39:24.333 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.26783/ceph-mon.a.asok config get mon_host 2021-08-21T14:39:24.333 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2021-08-21T14:39:24.333 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-prio 2021-08-21T14:39:24.333 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-08-21T14:39:24.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-08-21T14:39:24.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-08-21T14:39:24.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-prio/x 2021-08-21T14:39:24.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-08-21T14:39:24.866 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-08-21T14:39:24.867 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:39:24.867 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:39:24.867 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:39:24.868 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:24.868 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:24.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:39:24.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:39:24.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:39:24.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:335: TEST_backfill_pool_priority: export CEPH_ARGS 2021-08-21T14:39:24.877 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: expr 2 - 1 2021-08-21T14:39:24.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: seq 0 1 2021-08-21T14:39:24.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:39:24.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2021-08-21T14:39:24.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:39:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:39:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-08-21T14:39:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:39:24.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/0 2021-08-21T14:39:24.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:39:24.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:39:24.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:39:24.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:39:24.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2021-08-21T14:39:24.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2021-08-21T14:39:24.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:39:24.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:39:24.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:39:25.270 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:39:25.271 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:39:25.740 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:39:25.822 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:39:25.823 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:25.823 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:25.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:39:25.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:39:25.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:39:25.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:39:25.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:39:25.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:39:25.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:39:25.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:39:25.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:39:25.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:39:25.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:39:25.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:39:25.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/0 2021-08-21T14:39:25.826 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:39:26.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=a30feb2a-d757-448e-bd93-8db5822d70e7 2021-08-21T14:39:26.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 a30feb2a-d757-448e-bd93-8db5822d70e7' 2021-08-21T14:39:26.003 INFO:tasks.workunit.client.0.smithi077.stdout:add osd0 a30feb2a-d757-448e-bd93-8db5822d70e7 2021-08-21T14:39:26.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:39:26.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAZECFhWQZtCRAAnROb/o0KF+aIJN96xlY2Lw== 2021-08-21T14:39:26.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAZECFhWQZtCRAAnROb/o0KF+aIJN96xlY2Lw=="}' 2021-08-21T14:39:26.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new a30feb2a-d757-448e-bd93-8db5822d70e7 -i td/osd-backfill-prio/0/new.json 2021-08-21T14:39:26.878 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:39:26.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/0/new.json 2021-08-21T14:39:26.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAZECFhWQZtCRAAnROb/o0KF+aIJN96xlY2Lw== --osd-uuid a30feb2a-d757-448e-bd93-8db5822d70e7 2021-08-21T14:39:26.881 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.550+0000 7f9c0d42b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:26.881 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.551+0000 7f9c0d42b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:26.881 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.552+0000 7f9c0d42b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:26.882 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.553+0000 7f9c0d42b0c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-08-21T14:39:26.882 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.553+0000 7f9c0d42b0c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-08-21T14:39:26.882 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.553+0000 7f9c0d42b0c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-08-21T14:39:27.375 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:21.626+0000 7f9c0d42b0c0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2021-08-21T14:39:27.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2021-08-21T14:39:27.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:39:27.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-08-21T14:39:27.376 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd0 key to auth repository 2021-08-21T14:39:27.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:39:27.377 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.0 2021-08-21T14:39:27.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-08-21T14:39:27.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:39:27.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:39:27.667 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:39:27.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:39:27.758 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:27.108+0000 7f3869d410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:27.759 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:27.127+0000 7f3869d410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:27.759 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:27.128+0000 7f3869d410c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:27.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-08-21T14:39:27.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:39:27.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=0 2021-08-21T14:39:27.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:39:27.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:39:27.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:39:27.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:39:28.080 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:39:28.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:39:28.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:39:28.082 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:27.656+0000 7f3869d410c0 -1 Falling back to public interface 2021-08-21T14:39:28.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:39:28.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:39:28.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:39:28.881 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:39:28.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:39:28.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:39:28.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:39:29.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:39:29.973 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:29.958+0000 7f3869d410c0 -1 osd.0 0 log_to_monitors {default=true} 2021-08-21T14:39:30.356 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:39:30.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:39:30.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:39:30.357 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:39:30.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:39:30.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:39:30.797 INFO:tasks.workunit.client.0.smithi077.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/27908,v1:127.0.0.1:6803/27908] [v2:127.0.0.1:6804/27908,v1:127.0.0.1:6805/27908] exists,up a30feb2a-d757-448e-bd93-8db5822d70e7 2021-08-21T14:39:30.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:39:30.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:39:30.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:39:30.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:39:30.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2021-08-21T14:39:30.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:39:30.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:39:30.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-08-21T14:39:30.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:39:30.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/1 2021-08-21T14:39:30.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:39:30.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:39:30.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:39:30.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:39:30.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2021-08-21T14:39:30.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2021-08-21T14:39:30.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:39:30.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:39:30.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:39:30.963 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:39:30.963 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:39:30.963 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:39:30.963 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:39:30.964 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:39:30.964 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:39:30.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:39:30.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:39:31.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:39:31.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:39:31.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:39:31.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:39:31.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:39:31.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:39:31.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:39:31.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:39:31.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:39:31.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:39:31.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/1 2021-08-21T14:39:31.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:39:31.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=99fbbf2e-b90e-4f97-8482-1a322a7f5927 2021-08-21T14:39:31.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 99fbbf2e-b90e-4f97-8482-1a322a7f5927' 2021-08-21T14:39:31.826 INFO:tasks.workunit.client.0.smithi077.stdout:add osd1 99fbbf2e-b90e-4f97-8482-1a322a7f5927 2021-08-21T14:39:32.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:39:32.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAiECFhpYUpLxAAF4R11hGAZ3c2jx2ThE7LTw== 2021-08-21T14:39:32.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAiECFhpYUpLxAAF4R11hGAZ3c2jx2ThE7LTw=="}' 2021-08-21T14:39:32.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 99fbbf2e-b90e-4f97-8482-1a322a7f5927 -i td/osd-backfill-prio/1/new.json 2021-08-21T14:39:32.273 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:39:32.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/1/new.json 2021-08-21T14:39:32.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAiECFhpYUpLxAAF4R11hGAZ3c2jx2ThE7LTw== --osd-uuid 99fbbf2e-b90e-4f97-8482-1a322a7f5927 2021-08-21T14:39:32.372 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.383+0000 7f9e3e95c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:32.372 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.400+0000 7f9e3e95c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:32.373 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.400+0000 7f9e3e95c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:32.373 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.401+0000 7f9e3e95c0c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-08-21T14:39:32.373 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.401+0000 7f9e3e95c0c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-08-21T14:39:32.373 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.401+0000 7f9e3e95c0c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-08-21T14:39:32.373 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:31.492+0000 7f9e3e95c0c0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2021-08-21T14:39:36.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2021-08-21T14:39:36.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:39:36.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-08-21T14:39:36.856 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd1 key to auth repository 2021-08-21T14:39:36.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:39:37.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-08-21T14:39:37.418 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.1 2021-08-21T14:39:37.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:39:37.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:39:37.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:39:37.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:39:37.421 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:37.397+0000 7f8cd21690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:37.421 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:37.399+0000 7f8cd21690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:37.421 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:37.400+0000 7f8cd21690c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:39:37.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-08-21T14:39:37.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:39:37.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=1 2021-08-21T14:39:37.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:39:37.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:39:37.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:39:37.777 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:39:37.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:39:37.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:39:37.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:39:37.953 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:37.922+0000 7f8cd21690c0 -1 Falling back to public interface 2021-08-21T14:39:38.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:39:39.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:39:39.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:39:39.219 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:39:39.220 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:39:39.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:39:39.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:39:39.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:39:40.328 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:39:40.319+0000 7f8cd21690c0 -1 osd.1 0 log_to_monitors {default=true} 2021-08-21T14:39:40.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:39:40.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:39:40.636 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:39:40.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:39:40.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:39:40.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:39:41.147 INFO:tasks.workunit.client.0.smithi077.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29288,v1:127.0.0.1:6811/29288] [v2:127.0.0.1:6812/29288,v1:127.0.0.1:6813/29288] exists,up 99fbbf2e-b90e-4f97-8482-1a322a7f5927 2021-08-21T14:39:41.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:39:41.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:39:41.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:39:41.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: seq 1 3 2021-08-21T14:39:41.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:41.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test1 1 1 2021-08-21T14:39:41.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-08-21T14:39:41.719 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test1' already exists 2021-08-21T14:39:41.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:39:42.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2021-08-21T14:39:43.548 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 1 size to 2 2021-08-21T14:39:43.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:43.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test2 1 1 2021-08-21T14:39:43.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-08-21T14:39:44.114 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test2' already exists 2021-08-21T14:39:44.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:39:45.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2021-08-21T14:39:46.002 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 2 size to 2 2021-08-21T14:39:46.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:46.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test3 1 1 2021-08-21T14:39:46.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-08-21T14:39:46.613 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test3' already exists 2021-08-21T14:39:46.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:39:47.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2021-08-21T14:39:48.473 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 3 size to 2 2021-08-21T14:39:48.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:347: TEST_backfill_pool_priority: sleep 5 2021-08-21T14:39:53.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:349: TEST_backfill_pool_priority: wait_for_clean 2021-08-21T14:39:53.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:39:53.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:39:53.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:39:53.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:39:53.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:39:53.502 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:39:53.502 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:39:53.575 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:39:53.575 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:39:53.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:39:53.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:39:53.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:39:53.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:39:53.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:39:53.655 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:39:54.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:39:54.059 INFO:tasks.workunit.client.0.smithi077.stderr:1' 2021-08-21T14:39:54.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:39:54.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:39:54.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:39:54.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836488 2021-08-21T14:39:54.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836488 2021-08-21T14:39:54.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836488' 2021-08-21T14:39:54.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:39:54.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:39:54.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949672965 2021-08-21T14:39:54.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949672965 2021-08-21T14:39:54.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2021-08-21T14:39:54.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:39:54.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836488 2021-08-21T14:39:54.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:39:54.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:39:54.541 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836488 2021-08-21T14:39:54.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:39:54.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836488 2021-08-21T14:39:54.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2021-08-21T14:39:54.688 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836488 2021-08-21T14:39:54.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:39:54.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836486 -lt 21474836488 2021-08-21T14:39:54.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:39:55.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:39:55.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:39:56.231 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836488 -lt 21474836488 2021-08-21T14:39:56.232 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:39:56.232 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949672965 2021-08-21T14:39:56.232 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:39:56.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:39:56.233 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949672965 2021-08-21T14:39:56.233 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:39:56.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949672965 2021-08-21T14:39:56.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2021-08-21T14:39:56.332 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949672965 2021-08-21T14:39:56.332 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:39:56.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949672965 -lt 42949672965 2021-08-21T14:39:56.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:39:56.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:39:56.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:39:56.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 3 == 0 2021-08-21T14:39:56.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:39:56.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:39:56.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:39:56.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:39:56.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:39:57.091 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:39:57.092 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:39:57.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=3 2021-08-21T14:39:57.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:39:57.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:39:57.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:39:58.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 3 = 3 2021-08-21T14:39:58.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:39:58.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:39:58.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:351: TEST_backfill_pool_priority: ceph pg dump pgs 2021-08-21T14:39:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:39:58.243 INFO:tasks.workunit.client.0.smithi077.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:39:48.691255+0000 0'0 24:16 [1,0] 1 [1,0] 1 0'0 2021-08-21T14:39:46.327901+0000 0'0 2021-08-21T14:39:46.327901+0000 0 2021-08-21T14:39:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:39:43.700674+0000 0'0 23:26 [1,0] 1 [1,0] 1 0'0 2021-08-21T14:39:41.461452+0000 0'0 2021-08-21T14:39:41.461452+0000 0 2021-08-21T14:39:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:39:46.333664+0000 0'0 23:21 [0,1] 0 [0,1] 0 0'0 2021-08-21T14:39:43.878337+0000 0'0 2021-08-21T14:39:43.878337+0000 0 2021-08-21T14:39:58.244 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:39:58.244 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:39:58.246 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:39:58.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local PG1 2021-08-21T14:39:58.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local POOLNUM1 2021-08-21T14:39:58.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:357: TEST_backfill_pool_priority: local pool1 2021-08-21T14:39:58.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local chk_osd1_1 2021-08-21T14:39:58.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local chk_osd1_2 2021-08-21T14:39:58.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local PG2 2021-08-21T14:39:58.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local POOLNUM2 2021-08-21T14:39:58.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:363: TEST_backfill_pool_priority: local pool2 2021-08-21T14:39:58.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: local chk_osd2_1 2021-08-21T14:39:58.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:365: TEST_backfill_pool_priority: local chk_osd2_2 2021-08-21T14:39:58.799 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: seq 1 3 2021-08-21T14:39:58.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:58.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2021-08-21T14:39:58.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: jq '.acting[]' 2021-08-21T14:39:58.802 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:39:58.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: local test_osd1=1 2021-08-21T14:39:58.835 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:39:59.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: local test_osd2=0 2021-08-21T14:39:59.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: '[' -z '' ']' 2021-08-21T14:39:59.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: PG1=1.0 2021-08-21T14:39:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: POOLNUM1=1 2021-08-21T14:39:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: pool1=test1 2021-08-21T14:39:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:377: TEST_backfill_pool_priority: chk_osd1_1=1 2021-08-21T14:39:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: chk_osd1_2=0 2021-08-21T14:39:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:59.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2021-08-21T14:39:59.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: jq '.acting[]' 2021-08-21T14:39:59.216 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:39:59.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: local test_osd1=0 2021-08-21T14:39:59.217 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:39:59.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: local test_osd2=1 2021-08-21T14:39:59.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2021-08-21T14:39:59.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2021-08-21T14:39:59.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: PG2=2.0 2021-08-21T14:39:59.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: POOLNUM2=2 2021-08-21T14:39:59.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: pool2=test2 2021-08-21T14:39:59.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:384: TEST_backfill_pool_priority: chk_osd2_1=0 2021-08-21T14:39:59.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:385: TEST_backfill_pool_priority: chk_osd2_2=1 2021-08-21T14:39:59.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: break 2021-08-21T14:39:59.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:389: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2021-08-21T14:39:59.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:391: TEST_backfill_pool_priority: '[' test2 = '' ']' 2021-08-21T14:39:59.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: seq 1 3 2021-08-21T14:39:59.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:59.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2021-08-21T14:39:59.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:59.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2021-08-21T14:39:59.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2021-08-21T14:39:59.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2021-08-21T14:39:59.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:401: TEST_backfill_pool_priority: delete_pool test3 2021-08-21T14:39:59.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-08-21T14:39:59.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-08-21T14:40:00.185 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test3' does not exist 2021-08-21T14:40:00.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_extra_prio=1 2021-08-21T14:40:00.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_extra_prio=2 2021-08-21T14:40:00.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: expr 150 + 1 + 1 2021-08-21T14:40:00.186 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: pool1_prio=152 2021-08-21T14:40:00.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: expr 150 + 1 + 2 2021-08-21T14:40:00.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: pool2_prio=153 2021-08-21T14:40:00.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2021-08-21T14:40:00.499 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 1 size to 1 2021-08-21T14:40:00.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2021-08-21T14:40:01.494 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 1 recovery_priority to 1 2021-08-21T14:40:01.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:413: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2021-08-21T14:40:02.459 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 2 size to 1 2021-08-21T14:40:02.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2021-08-21T14:40:03.232 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 2 recovery_priority to 2 2021-08-21T14:40:03.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: wait_for_clean 2021-08-21T14:40:03.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:40:03.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:40:03.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:40:03.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:40:03.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:40:03.276 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:40:03.277 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:40:03.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:40:03.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:40:03.372 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:40:03.426 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:40:03.426 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:40:03.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:40:03.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:40:03.427 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:40:03.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:40:03.820 INFO:tasks.workunit.client.0.smithi077.stderr:1' 2021-08-21T14:40:03.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:40:03.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:40:04.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:40:04.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836492 2021-08-21T14:40:04.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836492 2021-08-21T14:40:04.442 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836492' 2021-08-21T14:40:04.442 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:40:04.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:40:04.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949672969 2021-08-21T14:40:04.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949672969 2021-08-21T14:40:04.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2021-08-21T14:40:04.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:40:04.694 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836492 2021-08-21T14:40:04.694 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:40:04.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:40:04.731 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836492 2021-08-21T14:40:04.731 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:40:04.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836492 2021-08-21T14:40:04.732 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836492 2021-08-21T14:40:04.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2021-08-21T14:40:04.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:40:04.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836490 -lt 21474836492 2021-08-21T14:40:04.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:40:05.431 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:40:05.431 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:40:05.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836492 -lt 21474836492 2021-08-21T14:40:05.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:40:05.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949672969 2021-08-21T14:40:05.857 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:40:05.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:40:05.857 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949672969 2021-08-21T14:40:05.857 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:40:05.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949672969 2021-08-21T14:40:06.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2021-08-21T14:40:06.189 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949672969 2021-08-21T14:40:06.190 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:40:06.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949672969 -lt 42949672969 2021-08-21T14:40:06.291 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:40:06.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:40:06.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:40:06.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 2 == 0 2021-08-21T14:40:06.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:40:06.826 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:40:06.841 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:40:06.842 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:40:06.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:40:06.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:40:06.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:40:07.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:40:07.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:40:07.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:40:07.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:40:07.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 2 2021-08-21T14:40:07.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:40:07.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:40:08.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:417: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2021-08-21T14:40:08.309 INFO:tasks.workunit.client.0.smithi077.stderr:10+0 records in 2021-08-21T14:40:08.309 INFO:tasks.workunit.client.0.smithi077.stderr:10+0 records out 2021-08-21T14:40:08.318 INFO:tasks.workunit.client.0.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.046239 s, 227 MB/s 2021-08-21T14:40:08.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: p=1 2021-08-21T14:40:08.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:419: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2021-08-21T14:40:08.811 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: seq 1 50 2021-08-21T14:40:08.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:08.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2021-08-21T14:40:08.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:08.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2021-08-21T14:40:08.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:08.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2021-08-21T14:40:08.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:08.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2021-08-21T14:40:09.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:09.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2021-08-21T14:40:09.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:09.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2021-08-21T14:40:09.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:10.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2021-08-21T14:40:10.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:10.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2021-08-21T14:40:10.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:10.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2021-08-21T14:40:10.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:10.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2021-08-21T14:40:11.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:11.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2021-08-21T14:40:11.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:11.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2021-08-21T14:40:12.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:12.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2021-08-21T14:40:12.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:12.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2021-08-21T14:40:12.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:12.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2021-08-21T14:40:13.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:13.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2021-08-21T14:40:13.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:13.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2021-08-21T14:40:13.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:13.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2021-08-21T14:40:14.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:14.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2021-08-21T14:40:14.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:14.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2021-08-21T14:40:14.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:14.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2021-08-21T14:40:15.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:15.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2021-08-21T14:40:15.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:15.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2021-08-21T14:40:15.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:15.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2021-08-21T14:40:16.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:16.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2021-08-21T14:40:16.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:16.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2021-08-21T14:40:16.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:16.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2021-08-21T14:40:17.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:17.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2021-08-21T14:40:17.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:17.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2021-08-21T14:40:18.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:18.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2021-08-21T14:40:18.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:18.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2021-08-21T14:40:18.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:18.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2021-08-21T14:40:19.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:19.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2021-08-21T14:40:19.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:19.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2021-08-21T14:40:19.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:19.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2021-08-21T14:40:20.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:20.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2021-08-21T14:40:20.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:20.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2021-08-21T14:40:20.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:20.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2021-08-21T14:40:21.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:21.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2021-08-21T14:40:21.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:21.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2021-08-21T14:40:21.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:21.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2021-08-21T14:40:22.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:22.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2021-08-21T14:40:22.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:22.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2021-08-21T14:40:22.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:22.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2021-08-21T14:40:23.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:23.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2021-08-21T14:40:23.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:23.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2021-08-21T14:40:24.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:24.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2021-08-21T14:40:24.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:24.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2021-08-21T14:40:24.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:24.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2021-08-21T14:40:25.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:25.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2021-08-21T14:40:25.515 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: expr 1 + 1 2021-08-21T14:40:25.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: p=2 2021-08-21T14:40:25.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:419: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2021-08-21T14:40:25.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: seq 1 50 2021-08-21T14:40:25.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:25.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2021-08-21T14:40:25.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:25.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2021-08-21T14:40:26.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:26.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2021-08-21T14:40:26.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:26.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2021-08-21T14:40:26.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:26.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2021-08-21T14:40:27.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:27.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2021-08-21T14:40:27.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:27.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2021-08-21T14:40:28.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:28.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2021-08-21T14:40:28.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:28.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2021-08-21T14:40:28.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:28.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2021-08-21T14:40:29.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:29.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2021-08-21T14:40:29.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:29.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2021-08-21T14:40:29.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:29.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2021-08-21T14:40:30.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:30.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2021-08-21T14:40:30.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:30.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2021-08-21T14:40:31.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:31.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2021-08-21T14:40:31.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:31.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2021-08-21T14:40:31.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:31.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2021-08-21T14:40:32.221 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:32.222 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2021-08-21T14:40:32.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:32.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2021-08-21T14:40:32.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:32.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2021-08-21T14:40:33.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:33.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2021-08-21T14:40:33.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:33.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2021-08-21T14:40:34.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:34.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2021-08-21T14:40:34.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:34.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2021-08-21T14:40:34.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:34.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2021-08-21T14:40:35.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:35.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2021-08-21T14:40:35.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:35.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2021-08-21T14:40:35.818 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:35.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2021-08-21T14:40:36.123 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:36.123 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2021-08-21T14:40:36.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:36.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2021-08-21T14:40:36.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:36.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2021-08-21T14:40:37.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:37.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2021-08-21T14:40:37.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:37.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2021-08-21T14:40:37.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:37.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2021-08-21T14:40:38.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:38.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2021-08-21T14:40:38.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:38.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2021-08-21T14:40:38.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:38.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2021-08-21T14:40:39.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:39.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2021-08-21T14:40:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2021-08-21T14:40:39.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:39.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2021-08-21T14:40:40.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:40.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2021-08-21T14:40:40.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:40.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2021-08-21T14:40:41.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:41.053 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2021-08-21T14:40:41.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:41.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2021-08-21T14:40:41.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:41.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2021-08-21T14:40:42.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:42.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2021-08-21T14:40:42.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:42.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2021-08-21T14:40:42.707 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:42.708 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2021-08-21T14:40:43.095 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2021-08-21T14:40:43.095 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2021-08-21T14:40:43.537 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: expr 2 + 1 2021-08-21T14:40:43.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: p=3 2021-08-21T14:40:43.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2021-08-21T14:40:43.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: get_not_primary: local poolname=test1 2021-08-21T14:40:43.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: get_not_primary: local objectname=obj1-p1 2021-08-21T14:40:43.546 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: get_not_primary: get_primary test1 obj1-p1 2021-08-21T14:40:43.546 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_primary: local poolname=test1 2021-08-21T14:40:43.546 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_primary: local objectname=obj1-p1 2021-08-21T14:40:43.549 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: get_primary: ceph --format json osd map test1 obj1-p1 2021-08-21T14:40:43.549 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: get_primary: jq .acting_primary 2021-08-21T14:40:43.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: get_not_primary: local primary=1 2021-08-21T14:40:43.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-08-21T14:40:43.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-08-21T14:40:44.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: local otherosd=null 2021-08-21T14:40:44.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph pg dump pgs 2021-08-21T14:40:44.758 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:40:44.758 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:40:00.628713+0000 34'150 34:191 [1] 1 [1] 1 0'0 2021-08-21T14:39:41.461452+0000 0'0 2021-08-21T14:39:41.461452+0000 0 2021-08-21T14:40:44.759 INFO:tasks.workunit.client.0.smithi077.stdout:2.0 44 0 0 0 0 459276288 0 0 31 31 active+clean 2021-08-21T14:40:02.296059+0000 34'131 34:168 [0] 0 [0] 0 0'0 2021-08-21T14:39:43.878337+0000 0'0 2021-08-21T14:39:43.878337+0000 0 2021-08-21T14:40:44.759 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:40:44.759 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:40:44.760 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:40:44.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ERRORS=0 2021-08-21T14:40:44.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2021-08-21T14:40:45.587 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 1 size to 2 2021-08-21T14:40:45.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2021-08-21T14:40:46.907 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 2 size to 2 2021-08-21T14:40:46.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: sleep 5 2021-08-21T14:40:51.928 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.1 2021-08-21T14:40:51.928 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:40:51.928 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:40:51.929 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:40:51.929 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:40:51.929 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:40:51.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:40:51.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2021-08-21T14:40:51.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:40:52.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.1 2021-08-21T14:40:52.049 INFO:tasks.workunit.client.0.smithi077.stdout:osd.1 2021-08-21T14:40:52.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-08-21T14:40:52.050 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:40:52.050 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:40:52.050 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:40:52.074 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:40:52.074 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:40:52.074 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:40:52.074 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:40:52.075 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:40:52.075 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 152, 2021-08-21T14:40:52.075 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:40:52.075 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:40:52.075 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:40:52.076 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:40:52.076 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:40:52.076 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:40:52.076 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:40:52.077 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:40:52.077 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:40:52.077 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:40:52.077 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "2.0", 2021-08-21T14:40:52.078 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 153, 2021-08-21T14:40:52.078 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:40:52.078 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:40:52.119 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:40:52.120 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:40:52.120 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:40:52.121 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: get_asok_path osd.0 2021-08-21T14:40:52.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2021-08-21T14:40:52.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2021-08-21T14:40:52.122 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:40:52.132 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:40:52.132 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:40:52.132 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.0.asok 2021-08-21T14:40:52.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: CEPH_ARGS= 2021-08-21T14:40:52.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.0.asok dump_recovery_reservations 2021-08-21T14:40:52.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:440: TEST_backfill_pool_priority: echo osd.0 2021-08-21T14:40:52.171 INFO:tasks.workunit.client.0.smithi077.stdout:osd.0 2021-08-21T14:40:52.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-08-21T14:40:52.172 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:40:52.172 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:40:52.173 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:40:52.173 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:40:52.173 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:40:52.173 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:40:52.173 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:40:52.174 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "2.0", 2021-08-21T14:40:52.174 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 153, 2021-08-21T14:40:52.174 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:40:52.254 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:40:52.254 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:40:52.254 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:40:52.302 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:40:52.312 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:40:52.312 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:40:52.312 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:40:52.313 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:40:52.409 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:40:52.409 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:40:52.409 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 152, 2021-08-21T14:40:52.409 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:40:52.410 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:40:52.410 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:40:52.410 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:40:52.471 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:40:52.599 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-08-21T14:40:52.599 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-08-21T14:40:52.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2021-08-21T14:40:52.600 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: ITEM=1.0 2021-08-21T14:40:52.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:445: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-08-21T14:40:52.600 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-08-21T14:40:52.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-08-21T14:40:52.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: PRIO=152 2021-08-21T14:40:52.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:451: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2021-08-21T14:40:52.702 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-08-21T14:40:52.702 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-08-21T14:40:52.910 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2021-08-21T14:40:52.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: ITEM=1.0 2021-08-21T14:40:52.911 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:460: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2021-08-21T14:40:52.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-08-21T14:40:52.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-08-21T14:40:52.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: PRIO=152 2021-08-21T14:40:52.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:466: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2021-08-21T14:40:52.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-08-21T14:40:52.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2021-08-21T14:40:52.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2021-08-21T14:40:52.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: ITEM=2.0 2021-08-21T14:40:52.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:475: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-08-21T14:40:52.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2021-08-21T14:40:52.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2021-08-21T14:40:53.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: PRIO=153 2021-08-21T14:40:53.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:481: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2021-08-21T14:40:53.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-08-21T14:40:53.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2021-08-21T14:40:53.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2021-08-21T14:40:53.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: ITEM=2.0 2021-08-21T14:40:53.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:490: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2021-08-21T14:40:53.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2021-08-21T14:40:53.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2021-08-21T14:40:53.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: PRIO=153 2021-08-21T14:40:53.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:496: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2021-08-21T14:40:53.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:503: TEST_backfill_pool_priority: wait_for_clean 2021-08-21T14:40:53.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:40:53.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:40:53.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:40:53.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:40:53.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:40:53.291 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:40:53.291 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:40:53.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:40:53.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:40:53.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:40:53.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:40:53.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:40:53.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:40:53.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:40:53.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:40:53.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:40:53.600 INFO:tasks.workunit.client.0.smithi077.stderr:1' 2021-08-21T14:40:53.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:40:53.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:40:53.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:40:53.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836504 2021-08-21T14:40:53.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836504 2021-08-21T14:40:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504' 2021-08-21T14:40:53.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:40:53.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:40:53.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949672981 2021-08-21T14:40:53.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949672981 2021-08-21T14:40:53.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504 1-42949672981' 2021-08-21T14:40:53.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:40:53.651 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836504 2021-08-21T14:40:53.651 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:40:53.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:40:53.678 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836504 2021-08-21T14:40:53.679 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:40:53.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836504 2021-08-21T14:40:53.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2021-08-21T14:40:53.680 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836504 2021-08-21T14:40:53.680 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:40:53.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836502 -lt 21474836504 2021-08-21T14:40:53.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:40:54.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:40:54.973 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:40:55.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836504 -lt 21474836504 2021-08-21T14:40:55.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:40:55.482 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949672981 2021-08-21T14:40:55.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:40:55.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:40:55.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949672981 2021-08-21T14:40:55.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:40:55.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949672981 2021-08-21T14:40:55.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2021-08-21T14:40:55.485 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949672981 2021-08-21T14:40:55.485 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:40:56.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949672981 -lt 42949672981 2021-08-21T14:40:56.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:40:56.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:40:56.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:40:56.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 2 == 0 2021-08-21T14:40:56.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:40:56.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:40:56.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:40:56.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:40:56.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:40:56.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:40:56.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:40:57.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:40:57.086 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:40:57.087 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:40:57.087 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:40:57.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:40:57.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' -1 2021-08-21T14:40:57.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: wait_for_clean: loop=0 2021-08-21T14:40:57.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: num_active_clean=0 2021-08-21T14:40:57.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:40:57.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:40:57.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:40:57.764 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:40:57.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:40:57.765 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:40:57.765 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:40:57.765 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:40:57.766 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:40:57.766 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:40:58.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:40:58.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:40:58.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:40:58.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:40:58.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:40:58.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:40:58.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:40:58.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:40:58.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:40:58.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:40:58.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:40:58.813 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:40:58.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:40:59.342 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24733765 2021-08-21T14:40:59.342 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24733765 '!=' null 2021-08-21T14:40:59.343 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:40:59.343 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:40:59.343 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:40:59.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:40:59.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:40:59.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:40:59.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:40:59.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:40:59.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:40:59.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:40:59.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:40:59.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:40:59.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:40:59.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:40:59.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:00.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:00.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:00.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:00.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:00.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:00.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:00.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:00.553 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:00.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:01.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=22714231 2021-08-21T14:41:01.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 22714231 '!=' null 2021-08-21T14:41:01.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:01.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:01.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:01.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:01.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:01.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:01.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:01.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:01.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:01.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:01.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:01.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:01.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:01.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:01.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:02.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:02.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:02.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:02.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:02.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:02.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:02.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:02.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:02.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:02.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=22714231 2021-08-21T14:41:02.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 22714231 '!=' null 2021-08-21T14:41:02.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:02.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:02.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:02.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:02.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:02.913 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:02.913 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:02.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:02.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:02.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:02.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:03.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:03.271 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:03.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:03.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:03.853 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:03.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:03.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:03.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:03.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:03.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:03.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:03.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:03.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:04.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=34945623 2021-08-21T14:41:04.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 34945623 '!=' null 2021-08-21T14:41:04.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:04.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:04.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:04.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:04.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:04.503 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:04.503 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:04.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:04.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:04.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:04.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:04.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:04.874 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:04.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:04.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:05.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:05.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:05.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:05.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:05.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:05.476 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:05.476 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:05.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:05.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:06.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24462367 2021-08-21T14:41:06.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24462367 '!=' null 2021-08-21T14:41:06.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:06.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:06.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:06.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:06.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:06.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:06.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:06.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:06.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:06.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:06.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:06.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:06.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:06.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:06.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:07.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:07.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:07.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:07.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:07.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:07.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:07.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:07.081 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:07.081 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:07.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=33199216 2021-08-21T14:41:07.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 33199216 '!=' null 2021-08-21T14:41:07.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:07.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:07.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:07.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:07.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:07.820 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:07.820 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:07.821 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:07.821 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:07.821 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:07.821 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:08.219 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:08.220 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:08.220 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:08.220 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:08.766 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:08.766 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:08.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:08.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:08.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:08.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:08.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:08.768 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:08.768 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:09.644 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24462711 2021-08-21T14:41:09.644 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24462711 '!=' null 2021-08-21T14:41:09.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:09.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:09.647 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:09.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:09.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:09.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:09.745 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:09.745 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:09.745 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:09.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:09.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:10.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:10.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:10.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:10.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:10.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:10.774 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:10.774 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:10.774 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:10.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:10.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:10.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:10.776 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:10.776 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:11.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24462636 2021-08-21T14:41:11.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24462636 '!=' null 2021-08-21T14:41:11.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:11.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:11.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:11.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:11.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:11.636 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:11.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:11.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:11.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:11.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:11.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:12.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:12.022 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:12.023 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:12.023 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:12.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:12.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:12.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:12.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:12.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:12.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:12.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:12.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:12.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:13.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=36693240 2021-08-21T14:41:13.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 36693240 '!=' null 2021-08-21T14:41:13.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:13.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:13.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:13.683 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:13.683 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:13.684 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:13.685 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:13.685 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:13.685 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:13.685 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:13.685 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:14.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:14.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:14.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:14.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:14.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:14.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:14.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:14.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:14.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:14.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:14.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:14.679 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:14.680 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:15.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=36693240 2021-08-21T14:41:15.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 36693240 '!=' null 2021-08-21T14:41:15.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:15.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:15.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:15.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:15.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:15.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:15.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:15.377 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:15.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:15.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:15.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:15.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:15.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:15.747 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:15.747 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:16.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:16.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:16.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:16.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:16.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:16.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:16.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:16.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:16.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:17.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=36693152 2021-08-21T14:41:17.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 36693152 '!=' null 2021-08-21T14:41:17.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:17.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:17.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:17.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:17.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:17.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:17.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:17.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:17.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:17.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:17.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:17.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:17.847 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:17.848 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:17.848 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:18.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:18.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:18.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:18.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:18.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:18.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:18.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:18.348 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:18.348 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:19.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=36693152 2021-08-21T14:41:19.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 36693152 '!=' null 2021-08-21T14:41:19.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:19.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:19.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:19.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:19.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:19.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:19.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:19.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:19.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:19.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:19.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:19.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:19.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:19.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:19.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:20.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:20.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:20.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:20.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:20.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:20.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:20.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:20.145 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:20.146 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:20.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24461927 2021-08-21T14:41:20.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24461927 '!=' null 2021-08-21T14:41:20.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:20.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:20.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:20.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:20.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:20.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:20.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:20.813 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:20.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:20.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:20.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:21.187 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:21.188 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:21.188 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:21.188 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:21.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:21.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:21.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:21.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:21.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:21.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:21.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:21.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:21.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:22.411 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24461552 2021-08-21T14:41:22.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24461552 '!=' null 2021-08-21T14:41:22.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:22.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:22.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:22.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:22.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:22.512 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:22.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:22.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:22.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:22.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:22.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:22.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:22.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:22.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:22.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:23.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:23.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:23.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:23.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:23.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:23.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:23.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:23.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:23.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:24.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=36693056 2021-08-21T14:41:24.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 36693056 '!=' null 2021-08-21T14:41:24.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:24.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:24.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:24.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:24.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:24.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:24.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:24.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:24.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:24.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:24.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:24.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:24.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:24.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:24.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:25.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:25.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:25.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:25.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:25.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:25.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:25.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:25.063 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:25.064 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:25.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=36694674 2021-08-21T14:41:25.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 36694674 '!=' null 2021-08-21T14:41:25.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:25.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:25.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:25.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:25.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:25.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:25.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:25.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:25.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:25.660 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:25.660 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:26.021 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:41:26.022 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:26.022 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:26.023 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:26.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 2 2021-08-21T14:41:26.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:41:26.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:41:26.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:41:26.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:41:26.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:41:26.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:41:26.515 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:41:26.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:41:27.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=9153586 2021-08-21T14:41:27.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 9153586 '!=' null 2021-08-21T14:41:27.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:41:27.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:41:27.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:41:27.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:41:27.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:41:27.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:41:27.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:41:27.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:41:27.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:41:27.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:41:27.110 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:41:27.476 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:41:27.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:41:27.477 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:41:27.477 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:41:27.970 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 2 2021-08-21T14:41:27.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:41:27.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:41:27.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:505: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2021-08-21T14:41:27.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: echo TEST PASSED 2021-08-21T14:41:27.972 INFO:tasks.workunit.client.0.smithi077.stdout:TEST PASSED 2021-08-21T14:41:27.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: delete_pool test1 2021-08-21T14:41:27.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-08-21T14:41:27.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-08-21T14:41:28.614 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test1' does not exist 2021-08-21T14:41:28.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:513: TEST_backfill_pool_priority: delete_pool test2 2021-08-21T14:41:28.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-08-21T14:41:28.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-08-21T14:41:29.192 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test2' does not exist 2021-08-21T14:41:29.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:514: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2021-08-21T14:41:29.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:41:29.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:41:29.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:41:29.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:41:29.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:41:29.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:41:29.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:515: TEST_backfill_pool_priority: return 0 2021-08-21T14:41:29.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:43: run: teardown td/osd-backfill-prio 2021-08-21T14:41:29.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-08-21T14:41:29.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:41:29.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-08-21T14:41:29.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:41:29.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:41:29.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:41:29.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:41:29.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:41:29.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:41:29.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:41:29.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:41:29.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:41:29.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:41:29.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:41:29.518 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:41:29.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:41:29.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:41:29.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:41:29.521 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:41:29.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:41:29.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:41:29.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-08-21T14:41:29.536 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:41:29.536 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:29.537 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:29.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26783 2021-08-21T14:41:29.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:41:29.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:41:29.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:40: run: for func in $funcs 2021-08-21T14:41:29.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:41: run: setup td/osd-backfill-prio 2021-08-21T14:41:29.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2021-08-21T14:41:29.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2021-08-21T14:41:29.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-08-21T14:41:29.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:41:29.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-08-21T14:41:29.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:41:29.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:41:29.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:41:29.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:41:29.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:41:29.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:41:29.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:41:29.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:41:29.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:41:29.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:41:29.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:41:29.544 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:41:29.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:41:29.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:41:29.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:41:29.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:41:29.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:41:29.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:41:29.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-08-21T14:41:29.548 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:41:29.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:29.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:29.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26783 2021-08-21T14:41:29.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:41:29.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:41:29.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2021-08-21T14:41:29.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-08-21T14:41:29.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:29.551 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:29.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26783 2021-08-21T14:41:29.551 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-08-21T14:41:29.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-08-21T14:41:29.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-08-21T14:41:29.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2021-08-21T14:41:29.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: TEST_backfill_priority td/osd-backfill-prio 2021-08-21T14:41:29.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:49: TEST_backfill_priority: local dir=td/osd-backfill-prio 2021-08-21T14:41:29.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:50: TEST_backfill_priority: local pools=10 2021-08-21T14:41:29.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:51: TEST_backfill_priority: local OSDS=5 2021-08-21T14:41:29.553 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:53: TEST_backfill_priority: expr 150 + 1 2021-08-21T14:41:29.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:53: TEST_backfill_priority: local degraded_prio=151 2021-08-21T14:41:29.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:54: TEST_backfill_priority: local max_tries=10 2021-08-21T14:41:29.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:56: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2021-08-21T14:41:29.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2021-08-21T14:41:29.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-08-21T14:41:29.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-08-21T14:41:29.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-08-21T14:41:29.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2021-08-21T14:41:29.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2021-08-21T14:41:29.902 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-08-21T14:41:29.903 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:41:29.903 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:41:29.903 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:41:29.904 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:29.904 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:29.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:29.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-08-21T14:41:30.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-08-21T14:41:30.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-08-21T14:41:30.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:41:30.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:41:30.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=fsid 2021-08-21T14:41:30.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .fsid 2021-08-21T14:41:30.118 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:41:30.118 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:41:30.118 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:41:30.119 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:41:30.119 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:30.119 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:30.119 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-mon.a.asok 2021-08-21T14:41:30.120 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:41:30.120 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.26783/ceph-mon.a.asok config get fsid 2021-08-21T14:41:30.215 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-08-21T14:41:30.215 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:41:30.215 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:41:30.216 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=mon_host 2021-08-21T14:41:30.216 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .mon_host 2021-08-21T14:41:30.216 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:41:30.216 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:41:30.217 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:41:30.217 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:41:30.217 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:30.217 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:30.217 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-mon.a.asok 2021-08-21T14:41:30.218 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:41:30.218 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.26783/ceph-mon.a.asok config get mon_host 2021-08-21T14:41:30.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:57: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2021-08-21T14:41:30.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-prio 2021-08-21T14:41:30.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-08-21T14:41:30.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-08-21T14:41:30.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-08-21T14:41:30.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-prio/x 2021-08-21T14:41:30.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-08-21T14:41:31.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-08-21T14:41:31.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:41:31.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:41:31.180 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:41:31.181 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:31.181 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:31.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:31.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:41:31.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:41:31.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:58: TEST_backfill_priority: export CEPH_ARGS 2021-08-21T14:41:31.207 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: expr 5 - 1 2021-08-21T14:41:31.213 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: seq 0 4 2021-08-21T14:41:31.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:41:31.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2021-08-21T14:41:31.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:41:31.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:41:31.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-08-21T14:41:31.215 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:41:31.215 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/0 2021-08-21T14:41:31.215 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:41:31.215 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:41:31.216 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:41:31.216 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:41:31.216 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2021-08-21T14:41:31.216 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2021-08-21T14:41:31.216 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:41:31.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:41:31.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:41:31.219 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:41:31.219 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:41:31.219 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:41:31.220 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:41:31.220 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:31.223 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:31.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:31.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:31.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:41:31.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:41:31.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:41:31.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:41:31.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:41:31.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:41:31.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:41:31.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:41:31.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:41:31.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:41:31.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/0 2021-08-21T14:41:31.229 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:41:31.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0cb31308-4325-40ff-8937-02f92c91a37a 2021-08-21T14:41:31.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 0cb31308-4325-40ff-8937-02f92c91a37a' 2021-08-21T14:41:31.230 INFO:tasks.workunit.client.0.smithi077.stdout:add osd0 0cb31308-4325-40ff-8937-02f92c91a37a 2021-08-21T14:41:31.230 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:41:31.231 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCbECFhDeW8DRAAYrX3H7uCHn9uo7T73DYyFQ== 2021-08-21T14:41:31.231 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCbECFhDeW8DRAAYrX3H7uCHn9uo7T73DYyFQ=="}' 2021-08-21T14:41:31.231 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0cb31308-4325-40ff-8937-02f92c91a37a -i td/osd-backfill-prio/0/new.json 2021-08-21T14:41:31.576 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:41:31.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/0/new.json 2021-08-21T14:41:31.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCbECFhDeW8DRAAYrX3H7uCHn9uo7T73DYyFQ== --osd-uuid 0cb31308-4325-40ff-8937-02f92c91a37a 2021-08-21T14:41:31.617 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.615+0000 7f337f0be0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:31.618 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.617+0000 7f337f0be0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:31.619 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.618+0000 7f337f0be0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:31.619 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.618+0000 7f337f0be0c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-08-21T14:41:31.620 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.618+0000 7f337f0be0c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-08-21T14:41:31.620 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.618+0000 7f337f0be0c0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2021-08-21T14:41:31.667 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:31.665+0000 7f337f0be0c0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2021-08-21T14:41:37.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2021-08-21T14:41:37.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:41:37.465 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd0 key to auth repository 2021-08-21T14:41:37.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-08-21T14:41:37.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:41:37.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-08-21T14:41:37.852 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.0 2021-08-21T14:41:37.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:41:37.853 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:41:37.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:41:37.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:41:37.878 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:37.874+0000 7f3e9dc7a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:37.879 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:37.876+0000 7f3e9dc7a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:37.879 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:37.876+0000 7f3e9dc7a0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:38.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-08-21T14:41:38.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:41:38.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=0 2021-08-21T14:41:38.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:41:38.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:41:38.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:38.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:41:38.237 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:41:38.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:38.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:41:38.407 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:38.407+0000 7f3e9dc7a0c0 -1 Falling back to public interface 2021-08-21T14:41:38.683 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:41:39.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:41:39.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:39.687 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:41:39.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:41:39.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:39.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:41:40.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:41:40.764 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:40.762+0000 7f3e9dc7a0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-08-21T14:41:41.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:41:41.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:41.075 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:41:41.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:41:41.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:41.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:41:41.446 INFO:tasks.workunit.client.0.smithi077.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/36115,v1:127.0.0.1:6803/36115] [v2:127.0.0.1:6804/36115,v1:127.0.0.1:6805/36115] exists,up 0cb31308-4325-40ff-8937-02f92c91a37a 2021-08-21T14:41:41.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:41:41.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:41:41.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:41:41.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:41:41.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2021-08-21T14:41:41.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:41:41.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:41:41.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-08-21T14:41:41.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:41:41.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/1 2021-08-21T14:41:41.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:41:41.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:41:41.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:41:41.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:41:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2021-08-21T14:41:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2021-08-21T14:41:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:41:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:41:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:41:41.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:41:41.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:41:41.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:41:41.509 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:41:41.519 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:41.520 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:41.520 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:41.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:41.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:41:41.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:41:41.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:41:41.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:41:41.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:41:41.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:41:41.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:41:41.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:41:41.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:41:41.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:41:41.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/1 2021-08-21T14:41:41.536 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:41:41.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5098db36-4f04-4184-9191-f7cadfd411b7 2021-08-21T14:41:41.536 INFO:tasks.workunit.client.0.smithi077.stdout:add osd1 5098db36-4f04-4184-9191-f7cadfd411b7 2021-08-21T14:41:41.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 5098db36-4f04-4184-9191-f7cadfd411b7' 2021-08-21T14:41:41.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:41:41.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQClECFhp4LkGxAA/LS4pnpEnMU7ZbCkYVONFQ== 2021-08-21T14:41:41.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQClECFhp4LkGxAA/LS4pnpEnMU7ZbCkYVONFQ=="}' 2021-08-21T14:41:41.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5098db36-4f04-4184-9191-f7cadfd411b7 -i td/osd-backfill-prio/1/new.json 2021-08-21T14:41:42.043 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:41:42.053 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/1/new.json 2021-08-21T14:41:42.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQClECFhp4LkGxAA/LS4pnpEnMU7ZbCkYVONFQ== --osd-uuid 5098db36-4f04-4184-9191-f7cadfd411b7 2021-08-21T14:41:42.086 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.079+0000 7f0ec0dc40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:42.105 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.100+0000 7f0ec0dc40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:42.106 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.101+0000 7f0ec0dc40c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:42.106 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.101+0000 7f0ec0dc40c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-08-21T14:41:42.107 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.101+0000 7f0ec0dc40c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-08-21T14:41:42.107 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.101+0000 7f0ec0dc40c0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2021-08-21T14:41:42.179 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:42.177+0000 7f0ec0dc40c0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2021-08-21T14:41:47.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2021-08-21T14:41:47.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:41:47.591 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd1 key to auth repository 2021-08-21T14:41:47.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-08-21T14:41:47.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:41:48.101 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.1 2021-08-21T14:41:48.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-08-21T14:41:48.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:41:48.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:41:48.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:41:48.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:41:48.129 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:48.125+0000 7f346a61e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:48.130 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:48.127+0000 7f346a61e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:48.130 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:48.128+0000 7f346a61e0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:48.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-08-21T14:41:48.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:41:48.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=1 2021-08-21T14:41:48.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:41:48.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:41:48.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:48.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:41:48.525 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:41:48.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:48.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:41:48.651 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:48.650+0000 7f346a61e0c0 -1 Falling back to public interface 2021-08-21T14:41:48.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:41:49.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:41:49.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:49.940 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:41:49.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:41:49.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:49.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:41:50.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:41:51.084 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:51.082+0000 7f346a61e0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-08-21T14:41:51.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:41:51.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:51.356 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:41:51.356 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:41:51.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:51.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:41:51.828 INFO:tasks.workunit.client.0.smithi077.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/37495,v1:127.0.0.1:6811/37495] [v2:127.0.0.1:6812/37495,v1:127.0.0.1:6813/37495] exists,up 5098db36-4f04-4184-9191-f7cadfd411b7 2021-08-21T14:41:51.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:41:51.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:41:51.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:41:51.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:41:51.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2021-08-21T14:41:51.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:41:51.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:41:51.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-08-21T14:41:51.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:41:51.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/2 2021-08-21T14:41:51.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:41:51.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:41:51.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:41:51.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:41:51.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2021-08-21T14:41:51.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2021-08-21T14:41:51.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:41:51.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:41:51.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:41:51.842 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:41:51.842 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:41:51.842 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:41:51.843 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:41:51.843 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:41:51.843 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:41:51.844 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:51.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:41:51.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:41:51.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:41:51.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:41:51.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:41:51.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:41:51.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:41:51.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:41:51.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:41:51.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:41:51.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:41:51.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/2 2021-08-21T14:41:51.849 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:41:51.849 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=0809eda2-97a3-4856-9994-a6611aa6631a 2021-08-21T14:41:51.850 INFO:tasks.workunit.client.0.smithi077.stdout:add osd2 0809eda2-97a3-4856-9994-a6611aa6631a 2021-08-21T14:41:51.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 0809eda2-97a3-4856-9994-a6611aa6631a' 2021-08-21T14:41:51.851 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:41:51.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCvECFh32SfMhAAdLoDvJ7AwGmX/Awzjw/FGA== 2021-08-21T14:41:51.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCvECFh32SfMhAAdLoDvJ7AwGmX/Awzjw/FGA=="}' 2021-08-21T14:41:51.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 0809eda2-97a3-4856-9994-a6611aa6631a -i td/osd-backfill-prio/2/new.json 2021-08-21T14:41:52.479 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:41:52.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/2/new.json 2021-08-21T14:41:52.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCvECFh32SfMhAAdLoDvJ7AwGmX/Awzjw/FGA== --osd-uuid 0809eda2-97a3-4856-9994-a6611aa6631a 2021-08-21T14:41:52.520 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.518+0000 7f16800490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:52.521 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.520+0000 7f16800490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:52.522 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.521+0000 7f16800490c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:52.523 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.521+0000 7f16800490c0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2021-08-21T14:41:52.523 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.521+0000 7f16800490c0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2021-08-21T14:41:52.524 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.521+0000 7f16800490c0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2021-08-21T14:41:52.568 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:52.567+0000 7f16800490c0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2021-08-21T14:41:57.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2021-08-21T14:41:57.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:41:57.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-08-21T14:41:57.960 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd2 key to auth repository 2021-08-21T14:41:57.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:41:58.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-08-21T14:41:58.465 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.2 2021-08-21T14:41:58.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:41:58.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:41:58.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:41:58.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:41:58.493 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:58.491+0000 7f67572740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:58.511 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:58.510+0000 7f67572740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:58.512 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:58.511+0000 7f67572740c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:41:58.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-08-21T14:41:58.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:41:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=2 2021-08-21T14:41:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:41:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:41:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:41:58.844 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:41:58.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:41:58.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:41:58.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:41:59.031 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:41:59.030+0000 7f67572740c0 -1 Falling back to public interface 2021-08-21T14:41:59.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:00.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:00.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:00.279 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:42:00.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:42:00.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:00.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:42:00.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:01.407 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:01.405+0000 7f67572740c0 -1 osd.2 0 log_to_monitors {default=true} 2021-08-21T14:42:01.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:01.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:01.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:42:01.747 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:42:01.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:01.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:42:02.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:03.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:03.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:03.160 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:42:03.162 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:42:03.162 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:03.162 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:42:03.588 INFO:tasks.workunit.client.0.smithi077.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/38883,v1:127.0.0.1:6819/38883] [v2:127.0.0.1:6820/38883,v1:127.0.0.1:6821/38883] exists,up 0809eda2-97a3-4856-9994-a6611aa6631a 2021-08-21T14:42:03.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:42:03.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:42:03.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:42:03.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:42:03.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2021-08-21T14:42:03.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:42:03.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:42:03.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-08-21T14:42:03.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:42:03.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/3 2021-08-21T14:42:03.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:42:03.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:42:03.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:42:03.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:42:03.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2021-08-21T14:42:03.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2021-08-21T14:42:03.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:42:03.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:42:03.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:42:03.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:42:03.598 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:42:03.598 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:42:03.598 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:42:03.600 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:42:03.600 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:42:03.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:42:03.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:42:03.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:42:03.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:42:03.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:42:03.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:42:03.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:42:03.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:42:03.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:42:03.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:42:03.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:42:03.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:42:03.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/3 2021-08-21T14:42:03.605 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:42:03.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=dd60af11-7bd7-49ce-b6c4-758a400235a6 2021-08-21T14:42:03.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 dd60af11-7bd7-49ce-b6c4-758a400235a6' 2021-08-21T14:42:03.606 INFO:tasks.workunit.client.0.smithi077.stdout:add osd3 dd60af11-7bd7-49ce-b6c4-758a400235a6 2021-08-21T14:42:03.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:42:03.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQC7ECFharZeJBAAsDt+H4Rmb4SF31zUAX+qdA== 2021-08-21T14:42:03.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQC7ECFharZeJBAAsDt+H4Rmb4SF31zUAX+qdA=="}' 2021-08-21T14:42:03.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new dd60af11-7bd7-49ce-b6c4-758a400235a6 -i td/osd-backfill-prio/3/new.json 2021-08-21T14:42:04.121 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:42:04.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/3/new.json 2021-08-21T14:42:04.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQC7ECFharZeJBAAsDt+H4Rmb4SF31zUAX+qdA== --osd-uuid dd60af11-7bd7-49ce-b6c4-758a400235a6 2021-08-21T14:42:04.163 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.162+0000 7f9d253b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:04.165 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.163+0000 7f9d253b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:04.165 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.164+0000 7f9d253b90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:04.167 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.164+0000 7f9d253b90c0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2021-08-21T14:42:04.167 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.164+0000 7f9d253b90c0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2021-08-21T14:42:04.168 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.164+0000 7f9d253b90c0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2021-08-21T14:42:04.237 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:04.236+0000 7f9d253b90c0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2021-08-21T14:42:10.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2021-08-21T14:42:10.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:42:10.003 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd3 key to auth repository 2021-08-21T14:42:10.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-08-21T14:42:10.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:42:10.519 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.3 2021-08-21T14:42:10.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-08-21T14:42:10.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:42:10.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:42:10.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:42:10.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:42:10.547 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:10.546+0000 7f9e1ac0b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:10.549 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:10.548+0000 7f9e1ac0b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:10.550 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:10.549+0000 7f9e1ac0b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:10.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-08-21T14:42:10.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:42:10.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=3 2021-08-21T14:42:10.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:42:10.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:42:10.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:10.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:42:10.937 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:42:10.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:10.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:42:11.066 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:11.065+0000 7f9e1ac0b0c0 -1 Falling back to public interface 2021-08-21T14:42:11.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:12.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:12.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:12.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:42:12.320 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:42:12.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:12.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:42:12.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:13.461 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:13.460+0000 7f9e1ac0b0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-08-21T14:42:13.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:13.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:13.749 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:42:13.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:42:13.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:13.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:42:14.261 INFO:tasks.workunit.client.0.smithi077.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/40289,v1:127.0.0.1:6827/40289] [v2:127.0.0.1:6828/40289,v1:127.0.0.1:6829/40289] exists,up dd60af11-7bd7-49ce-b6c4-758a400235a6 2021-08-21T14:42:14.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:42:14.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:42:14.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:42:14.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:42:14.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2021-08-21T14:42:14.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-prio 2021-08-21T14:42:14.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:42:14.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-08-21T14:42:14.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:42:14.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-prio/4 2021-08-21T14:42:14.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2021-08-21T14:42:14.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:42:14.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:42:14.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:42:14.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2021-08-21T14:42:14.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2021-08-21T14:42:14.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:42:14.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2021-08-21T14:42:14.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2021-08-21T14:42:14.275 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:42:14.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:42:14.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:42:14.276 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:42:14.276 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:42:14.277 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:42:14.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:42:14.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' 2021-08-21T14:42:14.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:42:14.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:42:14.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:42:14.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2021-08-21T14:42:14.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2021-08-21T14:42:14.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:42:14.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:42:14.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:42:14.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:42:14.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:42:14.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-prio/4 2021-08-21T14:42:14.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:42:14.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=002ede91-602a-44f6-a38d-68f7b573bb37 2021-08-21T14:42:14.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 002ede91-602a-44f6-a38d-68f7b573bb37' 2021-08-21T14:42:14.281 INFO:tasks.workunit.client.0.smithi077.stdout:add osd4 002ede91-602a-44f6-a38d-68f7b573bb37 2021-08-21T14:42:14.282 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:42:14.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDGECFhQ+BYEBAA8cuO+QbI/e9Ac06fs3nwug== 2021-08-21T14:42:14.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDGECFhQ+BYEBAA8cuO+QbI/e9Ac06fs3nwug=="}' 2021-08-21T14:42:14.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 002ede91-602a-44f6-a38d-68f7b573bb37 -i td/osd-backfill-prio/4/new.json 2021-08-21T14:42:14.858 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:42:14.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-prio/4/new.json 2021-08-21T14:42:14.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDGECFhQ+BYEBAA8cuO+QbI/e9Ac06fs3nwug== --osd-uuid 002ede91-602a-44f6-a38d-68f7b573bb37 2021-08-21T14:42:14.897 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.896+0000 7f0b6c1270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:14.898 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.898+0000 7f0b6c1270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:14.900 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.898+0000 7f0b6c1270c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:14.900 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.899+0000 7f0b6c1270c0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2021-08-21T14:42:14.901 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.899+0000 7f0b6c1270c0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2021-08-21T14:42:14.901 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.899+0000 7f0b6c1270c0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2021-08-21T14:42:14.963 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:14.961+0000 7f0b6c1270c0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2021-08-21T14:42:20.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2021-08-21T14:42:20.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:42:20.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2021-08-21T14:42:20.386 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd4 key to auth repository 2021-08-21T14:42:20.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:42:20.894 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.4 2021-08-21T14:42:20.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2021-08-21T14:42:20.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=51cfac0c-311f-4c5f-b20b-558fffb57ae2 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.26783/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:42:20.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:42:20.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:42:20.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:42:20.922 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:20.921+0000 7f34c8fec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:20.947 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:20.946+0000 7f34c8fec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:20.948 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:20.947+0000 7f34c8fec0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:42:21.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2021-08-21T14:42:21.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:42:21.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=4 2021-08-21T14:42:21.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:42:21.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:42:21.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:21.321 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:42:21.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:42:21.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:21.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:42:21.464 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:21.462+0000 7f34c8fec0c0 -1 Falling back to public interface 2021-08-21T14:42:21.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:22.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:22.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:22.697 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:42:22.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:42:22.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:22.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:42:23.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:23.900 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:42:23.880+0000 7f34c8fec0c0 -1 osd.4 0 log_to_monitors {default=true} 2021-08-21T14:42:24.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:24.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:24.079 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:42:24.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:42:24.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:24.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:42:24.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:42:25.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:42:25.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:42:25.501 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:42:25.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:42:25.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:42:25.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:42:25.884 INFO:tasks.workunit.client.0.smithi077.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/41669,v1:127.0.0.1:6835/41669] [v2:127.0.0.1:6836/41669,v1:127.0.0.1:6837/41669] exists,up 002ede91-602a-44f6-a38d-68f7b573bb37 2021-08-21T14:42:25.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:42:25.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:42:25.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:42:25.886 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: seq 1 10 2021-08-21T14:42:25.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:25.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test1 1 1 2021-08-21T14:42:25.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2021-08-21T14:42:26.487 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test1' already exists 2021-08-21T14:42:26.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:27.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test1 size 2 2021-08-21T14:42:28.279 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 1 size to 2 2021-08-21T14:42:28.288 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:28.288 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test2 1 1 2021-08-21T14:42:28.288 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2021-08-21T14:42:29.336 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test2' already exists 2021-08-21T14:42:29.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:30.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test2 size 2 2021-08-21T14:42:31.196 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 2 size to 2 2021-08-21T14:42:31.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:31.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test3 1 1 2021-08-21T14:42:31.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2021-08-21T14:42:32.084 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test3' already exists 2021-08-21T14:42:32.094 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:33.095 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test3 size 2 2021-08-21T14:42:33.911 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 3 size to 2 2021-08-21T14:42:33.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:33.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test4 1 1 2021-08-21T14:42:33.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2021-08-21T14:42:34.820 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test4' already exists 2021-08-21T14:42:34.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:35.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test4 size 2 2021-08-21T14:42:37.032 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 4 size to 2 2021-08-21T14:42:37.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:37.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test5 1 1 2021-08-21T14:42:37.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2021-08-21T14:42:37.756 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test5' already exists 2021-08-21T14:42:37.766 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:38.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test5 size 2 2021-08-21T14:42:39.411 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 5 size to 2 2021-08-21T14:42:39.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:39.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test6 1 1 2021-08-21T14:42:39.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2021-08-21T14:42:40.351 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test6' already exists 2021-08-21T14:42:40.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:41.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test6 size 2 2021-08-21T14:42:42.237 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 6 size to 2 2021-08-21T14:42:42.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:42.247 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test7 1 1 2021-08-21T14:42:42.247 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2021-08-21T14:42:43.080 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test7' already exists 2021-08-21T14:42:43.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:44.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test7 size 2 2021-08-21T14:42:45.224 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 7 size to 2 2021-08-21T14:42:45.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:45.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test8 1 1 2021-08-21T14:42:45.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2021-08-21T14:42:45.813 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test8' already exists 2021-08-21T14:42:45.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:46.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test8 size 2 2021-08-21T14:42:47.723 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 8 size to 2 2021-08-21T14:42:47.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:47.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test9 1 1 2021-08-21T14:42:47.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2021-08-21T14:42:48.397 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test9' already exists 2021-08-21T14:42:48.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:49.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test9 size 2 2021-08-21T14:42:50.414 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 9 size to 2 2021-08-21T14:42:50.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:42:50.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test10 1 1 2021-08-21T14:42:50.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2021-08-21T14:42:51.067 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test10' already exists 2021-08-21T14:42:51.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:42:52.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test10 size 2 2021-08-21T14:42:52.955 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 10 size to 2 2021-08-21T14:42:52.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:70: TEST_backfill_priority: sleep 5 2021-08-21T14:42:57.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:72: TEST_backfill_priority: wait_for_clean 2021-08-21T14:42:57.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:42:57.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:42:57.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:42:57.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:42:57.967 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:42:57.967 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:42:57.968 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:42:57.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:42:57.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:42:57.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:42:58.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:42:58.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:42:58.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:42:58.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:42:58.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:42:58.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:42:58.476 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:42:58.476 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:42:58.476 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:42:58.477 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:42:58.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:42:58.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:42:58.477 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:42:58.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836499 2021-08-21T14:42:58.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836499 2021-08-21T14:42:58.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836499' 2021-08-21T14:42:58.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:42:58.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:42:58.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949672976 2021-08-21T14:42:58.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949672976 2021-08-21T14:42:58.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836499 1-42949672976' 2021-08-21T14:42:58.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:42:58.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:42:58.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509454 2021-08-21T14:42:58.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509454 2021-08-21T14:42:58.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454' 2021-08-21T14:42:58.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:42:58.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:42:59.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345931 2021-08-21T14:42:59.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345931 2021-08-21T14:42:59.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931' 2021-08-21T14:42:59.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:42:59.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:42:59.175 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182410 2021-08-21T14:42:59.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182410 2021-08-21T14:42:59.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931 4-107374182410' 2021-08-21T14:42:59.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:42:59.177 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836499 2021-08-21T14:42:59.177 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:42:59.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:42:59.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836499 2021-08-21T14:42:59.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:42:59.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836499 2021-08-21T14:42:59.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2021-08-21T14:42:59.181 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836499 2021-08-21T14:42:59.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:42:59.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836497 -lt 21474836499 2021-08-21T14:42:59.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:43:00.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:43:00.594 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:43:01.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836499 -lt 21474836499 2021-08-21T14:43:01.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:01.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949672976 2021-08-21T14:43:01.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:01.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:43:01.030 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949672976 2021-08-21T14:43:01.030 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:01.031 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949672976 2021-08-21T14:43:01.031 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2021-08-21T14:43:01.031 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949672976 2021-08-21T14:43:01.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:43:01.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949672976 -lt 42949672976 2021-08-21T14:43:01.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:01.444 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509454 2021-08-21T14:43:01.444 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:01.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:43:01.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509454 2021-08-21T14:43:01.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:01.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509454 2021-08-21T14:43:01.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2021-08-21T14:43:01.447 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509454 2021-08-21T14:43:01.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:43:01.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509454 -lt 64424509454 2021-08-21T14:43:01.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:01.837 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345931 2021-08-21T14:43:01.837 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:01.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:43:01.838 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345931 2021-08-21T14:43:01.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:01.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345931 2021-08-21T14:43:01.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2021-08-21T14:43:01.840 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345931 2021-08-21T14:43:01.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:43:02.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345932 -lt 85899345931 2021-08-21T14:43:02.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:02.250 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182410 2021-08-21T14:43:02.250 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:02.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:43:02.252 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182410 2021-08-21T14:43:02.252 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:02.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182410 2021-08-21T14:43:02.254 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182410 2021-08-21T14:43:02.255 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2021-08-21T14:43:02.255 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:43:02.667 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182411 -lt 107374182410 2021-08-21T14:43:02.668 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:43:02.668 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:43:02.668 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:43:03.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 10 == 0 2021-08-21T14:43:03.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:43:03.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:43:03.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:43:03.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:43:03.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:43:03.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:43:03.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:43:03.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=10 2021-08-21T14:43:03.533 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:43:03.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:43:03.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:43:04.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 10 = 10 2021-08-21T14:43:04.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:43:04.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:43:04.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:74: TEST_backfill_priority: ceph pg dump pgs 2021-08-21T14:43:04.401 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:43:04.401 INFO:tasks.workunit.client.0.smithi077.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:53.539245+0000 0'0 81:20 [2,1] 2 [2,1] 2 0'0 2021-08-21T14:42:50.786510+0000 0'0 2021-08-21T14:42:50.786510+0000 0 2021-08-21T14:43:04.402 INFO:tasks.workunit.client.0.smithi077.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:51.409830+0000 0'0 80:17 [1,2] 1 [1,2] 1 0'0 2021-08-21T14:42:48.169494+0000 0'0 2021-08-21T14:42:48.169494+0000 0 2021-08-21T14:43:04.402 INFO:tasks.workunit.client.0.smithi077.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:48.266473+0000 0'0 80:26 [3,4] 3 [3,4] 3 0'0 2021-08-21T14:42:45.572490+0000 0'0 2021-08-21T14:42:45.572490+0000 0 2021-08-21T14:43:04.402 INFO:tasks.workunit.client.0.smithi077.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:45.992002+0000 0'0 80:31 [1,4] 1 [1,4] 1 0'0 2021-08-21T14:42:42.796484+0000 0'0 2021-08-21T14:42:42.796484+0000 0 2021-08-21T14:43:04.402 INFO:tasks.workunit.client.0.smithi077.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:32.201243+0000 0'0 80:53 [3,1] 3 [3,1] 3 0'0 2021-08-21T14:42:28.823964+0000 0'0 2021-08-21T14:42:28.823964+0000 0 2021-08-21T14:43:04.403 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:28.886875+0000 0'0 80:52 [1,0] 1 [1,0] 1 0'0 2021-08-21T14:42:26.190001+0000 0'0 2021-08-21T14:42:26.190001+0000 0 2021-08-21T14:43:04.403 INFO:tasks.workunit.client.0.smithi077.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:34.658208+0000 0'0 80:48 [1,2] 1 [1,2] 1 0'0 2021-08-21T14:42:31.714089+0000 0'0 2021-08-21T14:42:31.714089+0000 0 2021-08-21T14:43:04.403 INFO:tasks.workunit.client.0.smithi077.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:37.289937+0000 0'0 80:46 [4,0] 4 [4,0] 4 0'0 2021-08-21T14:42:34.415372+0000 0'0 2021-08-21T14:42:34.415372+0000 0 2021-08-21T14:43:04.403 INFO:tasks.workunit.client.0.smithi077.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:39.928614+0000 0'0 80:32 [3,4] 3 [3,4] 3 0'0 2021-08-21T14:42:37.465448+0000 0'0 2021-08-21T14:42:37.465448+0000 0 2021-08-21T14:43:04.404 INFO:tasks.workunit.client.0.smithi077.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2021-08-21T14:42:43.669594+0000 0'0 80:38 [4,2] 4 [4,2] 4 0'0 2021-08-21T14:42:39.835153+0000 0'0 2021-08-21T14:42:39.835153+0000 0 2021-08-21T14:43:04.404 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:43:04.404 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:43:04.404 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:43:04.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:78: TEST_backfill_priority: local PG1 2021-08-21T14:43:04.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:79: TEST_backfill_priority: local POOLNUM1 2021-08-21T14:43:04.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:80: TEST_backfill_priority: local pool1 2021-08-21T14:43:04.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:81: TEST_backfill_priority: local chk_osd1_1 2021-08-21T14:43:04.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:82: TEST_backfill_priority: local chk_osd1_2 2021-08-21T14:43:04.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:84: TEST_backfill_priority: local PG2 2021-08-21T14:43:04.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:85: TEST_backfill_priority: local POOLNUM2 2021-08-21T14:43:04.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:86: TEST_backfill_priority: local pool2 2021-08-21T14:43:04.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:87: TEST_backfill_priority: local chk_osd2 2021-08-21T14:43:04.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:89: TEST_backfill_priority: local PG3 2021-08-21T14:43:04.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:90: TEST_backfill_priority: local POOLNUM3 2021-08-21T14:43:04.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:91: TEST_backfill_priority: local pool3 2021-08-21T14:43:04.416 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: seq 1 10 2021-08-21T14:43:04.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:04.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 1.0 --format=json 2021-08-21T14:43:04.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:04.776 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:04.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2021-08-21T14:43:04.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:04.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=0 2021-08-21T14:43:04.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z '' ']' 2021-08-21T14:43:04.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:100: TEST_backfill_priority: PG1=1.0 2021-08-21T14:43:04.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:101: TEST_backfill_priority: POOLNUM1=1 2021-08-21T14:43:04.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:102: TEST_backfill_priority: pool1=test1 2021-08-21T14:43:04.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:103: TEST_backfill_priority: chk_osd1_1=1 2021-08-21T14:43:04.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:104: TEST_backfill_priority: chk_osd1_2=0 2021-08-21T14:43:04.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:04.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 2.0 --format=json 2021-08-21T14:43:04.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:05.164 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:05.165 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=3 2021-08-21T14:43:05.165 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:05.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=1 2021-08-21T14:43:05.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2021-08-21T14:43:05.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2021-08-21T14:43:05.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2021-08-21T14:43:05.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:05.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 3.0 --format=json 2021-08-21T14:43:05.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:05.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:05.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2021-08-21T14:43:05.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=2 2021-08-21T14:43:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2021-08-21T14:43:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2021-08-21T14:43:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:107: TEST_backfill_priority: PG2=3.0 2021-08-21T14:43:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:108: TEST_backfill_priority: POOLNUM2=3 2021-08-21T14:43:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:109: TEST_backfill_priority: pool2=test3 2021-08-21T14:43:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:110: TEST_backfill_priority: chk_osd2=2 2021-08-21T14:43:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:05.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 4.0 --format=json 2021-08-21T14:43:05.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:05.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:05.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=4 2021-08-21T14:43:05.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:05.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=0 2021-08-21T14:43:05.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2021-08-21T14:43:05.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2021-08-21T14:43:05.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2021-08-21T14:43:05.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:05.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 5.0 --format=json 2021-08-21T14:43:05.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:06.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:06.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=3 2021-08-21T14:43:06.333 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:06.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=4 2021-08-21T14:43:06.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2021-08-21T14:43:06.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2021-08-21T14:43:06.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-08-21T14:43:06.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:06.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 6.0 --format=json 2021-08-21T14:43:06.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:06.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:06.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=4 2021-08-21T14:43:06.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:06.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=2 2021-08-21T14:43:06.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2021-08-21T14:43:06.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2021-08-21T14:43:06.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2021-08-21T14:43:06.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:06.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 7.0 --format=json 2021-08-21T14:43:06.724 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2021-08-21T14:43:07.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2021-08-21T14:43:07.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2021-08-21T14:43:07.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2021-08-21T14:43:07.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=4 2021-08-21T14:43:07.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2021-08-21T14:43:07.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2021-08-21T14:43:07.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2021-08-21T14:43:07.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:113: TEST_backfill_priority: PG3=7.0 2021-08-21T14:43:07.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:114: TEST_backfill_priority: POOLNUM3=7 2021-08-21T14:43:07.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:115: TEST_backfill_priority: pool3=test7 2021-08-21T14:43:07.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:116: TEST_backfill_priority: break 2021-08-21T14:43:07.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:119: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2021-08-21T14:43:07.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:121: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2021-08-21T14:43:07.128 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: seq 1 10 2021-08-21T14:43:07.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:07.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2021-08-21T14:43:07.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:07.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2021-08-21T14:43:07.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test2 2021-08-21T14:43:07.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2021-08-21T14:43:07.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2021-08-21T14:43:07.718 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test2' does not exist 2021-08-21T14:43:07.727 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:07.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2021-08-21T14:43:07.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:07.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2021-08-21T14:43:07.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test4 2021-08-21T14:43:07.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2021-08-21T14:43:07.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2021-08-21T14:43:08.289 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test4' does not exist 2021-08-21T14:43:08.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:08.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2021-08-21T14:43:08.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test5 2021-08-21T14:43:08.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2021-08-21T14:43:08.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2021-08-21T14:43:08.946 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test5' does not exist 2021-08-21T14:43:08.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:08.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2021-08-21T14:43:08.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test6 2021-08-21T14:43:08.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2021-08-21T14:43:08.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2021-08-21T14:43:09.585 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test6' does not exist 2021-08-21T14:43:09.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:09.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2021-08-21T14:43:09.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:09.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2021-08-21T14:43:09.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test8 2021-08-21T14:43:09.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2021-08-21T14:43:09.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2021-08-21T14:43:10.331 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test8' does not exist 2021-08-21T14:43:10.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:10.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2021-08-21T14:43:10.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test9 2021-08-21T14:43:10.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2021-08-21T14:43:10.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2021-08-21T14:43:10.944 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test9' does not exist 2021-08-21T14:43:10.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2021-08-21T14:43:10.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2021-08-21T14:43:10.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test10 2021-08-21T14:43:10.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2021-08-21T14:43:10.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2021-08-21T14:43:11.503 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test10' does not exist 2021-08-21T14:43:11.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:135: TEST_backfill_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2021-08-21T14:43:12.502 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 3 size to 1 2021-08-21T14:43:12.510 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:136: TEST_backfill_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2021-08-21T14:43:13.169 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 7 size to 1 2021-08-21T14:43:13.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:137: TEST_backfill_priority: wait_for_clean 2021-08-21T14:43:13.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:43:13.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:43:13.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:43:13.181 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:43:13.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:43:13.181 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:43:13.181 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:43:13.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:43:13.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:43:13.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:43:13.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:43:13.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:43:13.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:43:13.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:43:13.270 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:43:13.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:43:13.673 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:43:13.673 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:43:13.673 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:43:13.674 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:43:13.674 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:43:13.674 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:43:13.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:43:13.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836504 2021-08-21T14:43:13.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836504 2021-08-21T14:43:13.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504' 2021-08-21T14:43:13.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:43:13.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:43:14.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949672981 2021-08-21T14:43:14.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949672981 2021-08-21T14:43:14.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504 1-42949672981' 2021-08-21T14:43:14.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:43:14.062 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:43:14.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509459 2021-08-21T14:43:14.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509459 2021-08-21T14:43:14.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504 1-42949672981 2-64424509459' 2021-08-21T14:43:14.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:43:14.200 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:43:14.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345936 2021-08-21T14:43:14.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345936 2021-08-21T14:43:14.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504 1-42949672981 2-64424509459 3-85899345936' 2021-08-21T14:43:14.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:43:14.358 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:43:14.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182415 2021-08-21T14:43:14.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182415 2021-08-21T14:43:14.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836504 1-42949672981 2-64424509459 3-85899345936 4-107374182415' 2021-08-21T14:43:14.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:14.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836504 2021-08-21T14:43:14.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:14.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:43:14.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836504 2021-08-21T14:43:14.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:14.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836504 2021-08-21T14:43:14.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2021-08-21T14:43:14.503 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836504 2021-08-21T14:43:14.503 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:43:14.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836502 -lt 21474836504 2021-08-21T14:43:14.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:43:15.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:43:15.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:43:16.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836504 -lt 21474836504 2021-08-21T14:43:16.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:16.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949672981 2021-08-21T14:43:16.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:16.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:43:16.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949672981 2021-08-21T14:43:16.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:16.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949672981 2021-08-21T14:43:16.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2021-08-21T14:43:16.300 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949672981 2021-08-21T14:43:16.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:43:16.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949672981 -lt 42949672981 2021-08-21T14:43:16.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:16.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509459 2021-08-21T14:43:16.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:16.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:43:16.690 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509459 2021-08-21T14:43:16.690 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:16.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509459 2021-08-21T14:43:16.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2021-08-21T14:43:16.691 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509459 2021-08-21T14:43:16.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:43:17.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509459 -lt 64424509459 2021-08-21T14:43:17.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:17.104 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345936 2021-08-21T14:43:17.104 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:17.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:43:17.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345936 2021-08-21T14:43:17.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:17.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345936 2021-08-21T14:43:17.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345936' 2021-08-21T14:43:17.106 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345936 2021-08-21T14:43:17.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:43:17.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345937 -lt 85899345936 2021-08-21T14:43:17.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:43:17.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182415 2021-08-21T14:43:17.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:43:17.563 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:43:17.563 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182415 2021-08-21T14:43:17.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:43:17.564 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182415 2021-08-21T14:43:17.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2021-08-21T14:43:17.565 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182415 2021-08-21T14:43:17.565 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:43:17.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182416 -lt 107374182415 2021-08-21T14:43:17.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:43:17.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:43:17.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:43:18.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 3 == 0 2021-08-21T14:43:18.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:43:18.487 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:43:18.487 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:43:18.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:43:18.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:43:18.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:43:18.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:43:18.870 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=3 2021-08-21T14:43:18.870 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:43:18.871 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:43:18.871 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:43:19.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 3 = 3 2021-08-21T14:43:19.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:43:19.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:43:19.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:139: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2021-08-21T14:43:19.406 INFO:tasks.workunit.client.0.smithi077.stderr:10+0 records in 2021-08-21T14:43:19.406 INFO:tasks.workunit.client.0.smithi077.stderr:10+0 records out 2021-08-21T14:43:19.406 INFO:tasks.workunit.client.0.smithi077.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0462639 s, 227 MB/s 2021-08-21T14:43:19.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:140: TEST_backfill_priority: p=1 2021-08-21T14:43:19.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2021-08-21T14:43:19.407 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2021-08-21T14:43:19.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:19.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2021-08-21T14:43:20.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:20.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2021-08-21T14:43:20.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:20.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2021-08-21T14:43:21.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:21.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2021-08-21T14:43:21.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:21.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2021-08-21T14:43:22.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:22.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2021-08-21T14:43:23.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:23.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2021-08-21T14:43:23.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:23.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2021-08-21T14:43:24.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:24.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2021-08-21T14:43:25.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:25.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2021-08-21T14:43:25.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:25.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2021-08-21T14:43:26.390 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:26.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2021-08-21T14:43:27.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:27.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2021-08-21T14:43:27.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:27.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2021-08-21T14:43:28.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:28.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2021-08-21T14:43:28.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:28.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2021-08-21T14:43:29.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:29.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2021-08-21T14:43:30.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:30.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2021-08-21T14:43:30.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:30.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2021-08-21T14:43:31.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:31.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2021-08-21T14:43:32.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:32.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2021-08-21T14:43:32.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:32.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2021-08-21T14:43:33.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:33.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2021-08-21T14:43:33.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:33.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2021-08-21T14:43:34.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:34.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2021-08-21T14:43:35.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:35.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2021-08-21T14:43:35.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:35.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2021-08-21T14:43:36.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:36.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2021-08-21T14:43:37.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:37.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2021-08-21T14:43:37.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:37.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2021-08-21T14:43:38.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:38.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2021-08-21T14:43:39.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:39.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2021-08-21T14:43:39.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:39.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2021-08-21T14:43:40.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:40.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2021-08-21T14:43:41.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:41.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2021-08-21T14:43:41.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:41.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2021-08-21T14:43:42.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:42.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2021-08-21T14:43:43.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:43.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2021-08-21T14:43:43.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:43.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2021-08-21T14:43:44.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:44.390 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2021-08-21T14:43:45.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:45.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2021-08-21T14:43:45.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:45.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2021-08-21T14:43:46.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:46.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2021-08-21T14:43:46.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:46.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2021-08-21T14:43:47.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:47.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2021-08-21T14:43:48.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:48.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2021-08-21T14:43:48.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:48.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2021-08-21T14:43:49.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:49.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2021-08-21T14:43:50.223 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:50.223 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2021-08-21T14:43:50.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:50.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2021-08-21T14:43:51.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 1 + 1 2021-08-21T14:43:51.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=2 2021-08-21T14:43:51.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2021-08-21T14:43:51.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2021-08-21T14:43:51.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:51.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2021-08-21T14:43:51.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:51.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2021-08-21T14:43:52.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:52.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2021-08-21T14:43:52.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:52.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2021-08-21T14:43:52.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:52.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2021-08-21T14:43:53.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:53.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2021-08-21T14:43:53.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:53.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2021-08-21T14:43:54.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:54.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2021-08-21T14:43:54.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:54.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2021-08-21T14:43:54.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:54.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2021-08-21T14:43:55.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:55.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2021-08-21T14:43:55.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:55.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2021-08-21T14:43:55.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:55.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2021-08-21T14:43:56.308 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:56.308 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2021-08-21T14:43:56.708 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:56.709 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2021-08-21T14:43:57.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:57.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2021-08-21T14:43:57.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:57.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2021-08-21T14:43:57.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:57.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2021-08-21T14:43:58.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:58.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2021-08-21T14:43:58.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:58.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2021-08-21T14:43:58.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:58.784 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2021-08-21T14:43:59.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:59.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2021-08-21T14:43:59.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:59.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2021-08-21T14:43:59.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:43:59.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2021-08-21T14:44:00.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:00.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2021-08-21T14:44:00.608 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:00.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2021-08-21T14:44:00.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:00.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2021-08-21T14:44:01.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:01.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2021-08-21T14:44:01.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:01.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2021-08-21T14:44:02.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:02.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2021-08-21T14:44:02.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:02.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2021-08-21T14:44:02.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:02.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2021-08-21T14:44:03.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:03.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2021-08-21T14:44:03.395 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:03.395 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2021-08-21T14:44:03.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:03.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2021-08-21T14:44:04.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:04.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2021-08-21T14:44:04.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:04.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2021-08-21T14:44:04.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:04.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2021-08-21T14:44:05.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:05.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2021-08-21T14:44:05.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:05.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2021-08-21T14:44:06.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:06.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2021-08-21T14:44:06.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:06.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2021-08-21T14:44:06.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:06.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2021-08-21T14:44:07.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:07.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2021-08-21T14:44:07.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:07.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2021-08-21T14:44:07.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:07.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2021-08-21T14:44:08.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:08.098 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2021-08-21T14:44:08.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:08.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2021-08-21T14:44:08.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:08.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2021-08-21T14:44:09.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:09.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2021-08-21T14:44:09.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 2 + 1 2021-08-21T14:44:09.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=3 2021-08-21T14:44:09.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2021-08-21T14:44:09.614 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2021-08-21T14:44:09.615 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:09.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2021-08-21T14:44:10.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:10.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2021-08-21T14:44:10.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:10.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2021-08-21T14:44:10.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:10.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2021-08-21T14:44:11.094 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:11.095 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2021-08-21T14:44:11.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:11.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2021-08-21T14:44:11.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:11.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2021-08-21T14:44:12.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:12.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2021-08-21T14:44:12.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:12.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2021-08-21T14:44:12.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:12.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2021-08-21T14:44:13.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:13.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2021-08-21T14:44:13.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:13.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2021-08-21T14:44:13.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:13.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2021-08-21T14:44:14.432 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:14.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2021-08-21T14:44:14.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:14.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2021-08-21T14:44:15.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:15.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2021-08-21T14:44:15.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:15.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2021-08-21T14:44:15.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:15.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2021-08-21T14:44:16.223 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:16.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2021-08-21T14:44:16.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:16.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2021-08-21T14:44:17.018 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:17.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2021-08-21T14:44:17.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:17.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2021-08-21T14:44:17.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:17.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2021-08-21T14:44:18.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:18.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2021-08-21T14:44:18.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:18.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2021-08-21T14:44:18.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:18.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2021-08-21T14:44:19.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:19.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2021-08-21T14:44:19.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:19.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2021-08-21T14:44:19.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:19.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2021-08-21T14:44:20.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:20.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2021-08-21T14:44:20.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:20.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2021-08-21T14:44:21.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:21.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2021-08-21T14:44:21.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:21.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2021-08-21T14:44:21.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:21.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2021-08-21T14:44:22.098 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:22.098 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2021-08-21T14:44:22.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:22.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2021-08-21T14:44:22.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:22.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2021-08-21T14:44:23.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:23.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2021-08-21T14:44:23.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:23.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2021-08-21T14:44:23.947 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:23.947 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2021-08-21T14:44:24.390 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:24.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2021-08-21T14:44:24.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:24.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2021-08-21T14:44:25.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:25.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2021-08-21T14:44:25.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:25.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2021-08-21T14:44:25.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:25.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2021-08-21T14:44:26.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:26.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2021-08-21T14:44:26.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:26.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2021-08-21T14:44:26.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:26.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2021-08-21T14:44:27.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:27.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2021-08-21T14:44:27.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2021-08-21T14:44:27.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2021-08-21T14:44:27.989 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 3 + 1 2021-08-21T14:44:27.990 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=4 2021-08-21T14:44:27.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:150: TEST_backfill_priority: get_not_primary test1 obj1-p1 2021-08-21T14:44:27.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: get_not_primary: local poolname=test1 2021-08-21T14:44:27.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: get_not_primary: local objectname=obj1-p1 2021-08-21T14:44:27.992 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: get_not_primary: get_primary test1 obj1-p1 2021-08-21T14:44:27.992 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: get_primary: local poolname=test1 2021-08-21T14:44:27.992 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: get_primary: local objectname=obj1-p1 2021-08-21T14:44:27.992 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: get_primary: ceph --format json osd map test1 obj1-p1 2021-08-21T14:44:27.992 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: get_primary: jq .acting_primary 2021-08-21T14:44:28.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: get_not_primary: local primary=1 2021-08-21T14:44:28.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: get_not_primary: ceph --format json osd map test1 obj1-p1 2021-08-21T14:44:28.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2021-08-21T14:44:28.801 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:150: TEST_backfill_priority: local otherosd=0 2021-08-21T14:44:28.802 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph pg dump pgs 2021-08-21T14:44:29.154 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:44:29.155 INFO:tasks.workunit.client.0.smithi077.stdout:7.0 49 0 0 0 0 511705088 0 0 46 46 active+clean 2021-08-21T14:43:13.938545+0000 93'146 93:189 [1] 1 [1] 1 0'0 2021-08-21T14:42:42.796484+0000 0'0 2021-08-21T14:42:42.796484+0000 0 2021-08-21T14:44:29.155 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:42:28.886875+0000 93'150 93:207 [1,0] 1 [1,0] 1 0'0 2021-08-21T14:42:26.190001+0000 0'0 2021-08-21T14:42:26.190001+0000 0 2021-08-21T14:44:29.155 INFO:tasks.workunit.client.0.smithi077.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:43:14.001473+0000 93'150 93:210 [1] 1 [1] 1 0'0 2021-08-21T14:42:31.714089+0000 0'0 2021-08-21T14:42:31.714089+0000 0 2021-08-21T14:44:29.155 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:44:29.156 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:44:29.156 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:44:29.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:153: TEST_backfill_priority: ERRORS=0 2021-08-21T14:44:29.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:155: TEST_backfill_priority: ceph osd set nobackfill 2021-08-21T14:44:29.921 INFO:tasks.workunit.client.0.smithi077.stderr:nobackfill is set 2021-08-21T14:44:29.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:156: TEST_backfill_priority: ceph osd set noout 2021-08-21T14:44:30.758 INFO:tasks.workunit.client.0.smithi077.stderr:noout is set 2021-08-21T14:44:30.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph osd pool set test7 size 2 2021-08-21T14:44:31.562 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 7 size to 2 2021-08-21T14:44:31.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:161: TEST_backfill_priority: sleep 2 2021-08-21T14:44:33.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:44:33.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:44:33.575 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:44:33.575 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:44:33.575 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:44:33.575 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:44:33.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:44:33.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:44:33.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:44:33.679 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:44:33.679 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:44:33.680 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:33.680 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:33.680 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:33.680 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:44:33.681 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:33.681 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:44:33.681 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:44:33.681 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:33.681 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:33.682 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:33.682 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:33.682 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:44:33.682 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:33.683 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:33.683 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:33.683 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:44:33.683 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:33.683 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:44:33.688 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: seq 1 10 2021-08-21T14:44:33.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-08-21T14:44:33.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-08-21T14:44:33.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-08-21T14:44:34.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:171: TEST_backfill_priority: '[' 1 = 10 ']' 2021-08-21T14:44:34.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:175: TEST_backfill_priority: sleep 2 2021-08-21T14:44:36.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-08-21T14:44:36.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-08-21T14:44:36.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-08-21T14:44:36.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:171: TEST_backfill_priority: '[' 2 = 10 ']' 2021-08-21T14:44:36.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:175: TEST_backfill_priority: sleep 2 2021-08-21T14:44:38.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-08-21T14:44:38.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-08-21T14:44:38.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-08-21T14:44:38.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:169: TEST_backfill_priority: break 2021-08-21T14:44:38.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:177: TEST_backfill_priority: flush_pg_stats 2021-08-21T14:44:38.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:44:38.819 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:44:39.232 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:44:39.233 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:44:39.233 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:44:39.233 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:44:39.234 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:44:39.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:44:39.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:39.234 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:44:39.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836523 2021-08-21T14:44:39.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836523 2021-08-21T14:44:39.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836523' 2021-08-21T14:44:39.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:39.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:44:39.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673000 2021-08-21T14:44:39.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673000 2021-08-21T14:44:39.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836523 1-42949673000' 2021-08-21T14:44:39.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:39.505 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:44:39.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509478 2021-08-21T14:44:39.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509478 2021-08-21T14:44:39.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836523 1-42949673000 2-64424509478' 2021-08-21T14:44:39.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:39.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:44:39.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345955 2021-08-21T14:44:39.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345955 2021-08-21T14:44:39.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836523 1-42949673000 2-64424509478 3-85899345955' 2021-08-21T14:44:39.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:39.776 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:44:39.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182434 2021-08-21T14:44:39.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182434 2021-08-21T14:44:39.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836523 1-42949673000 2-64424509478 3-85899345955 4-107374182434' 2021-08-21T14:44:39.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:39.943 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836523 2021-08-21T14:44:39.943 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:39.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:44:39.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836523 2021-08-21T14:44:39.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:39.945 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836523 2021-08-21T14:44:39.946 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836523 2021-08-21T14:44:39.946 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2021-08-21T14:44:39.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:44:40.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836523 -lt 21474836523 2021-08-21T14:44:40.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:40.351 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673000 2021-08-21T14:44:40.352 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:40.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:44:40.353 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673000 2021-08-21T14:44:40.353 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:40.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673000 2021-08-21T14:44:40.354 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673000 2021-08-21T14:44:40.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2021-08-21T14:44:40.355 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:44:40.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673000 -lt 42949673000 2021-08-21T14:44:40.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:40.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509478 2021-08-21T14:44:40.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:40.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:44:40.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509478 2021-08-21T14:44:40.772 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:40.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509478 2021-08-21T14:44:40.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2021-08-21T14:44:40.772 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509478 2021-08-21T14:44:40.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:44:41.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509478 -lt 64424509478 2021-08-21T14:44:41.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:41.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345955 2021-08-21T14:44:41.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:41.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:44:41.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345955 2021-08-21T14:44:41.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:41.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345955 2021-08-21T14:44:41.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2021-08-21T14:44:41.207 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345955 2021-08-21T14:44:41.207 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:44:41.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345956 -lt 85899345955 2021-08-21T14:44:41.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:41.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182434 2021-08-21T14:44:41.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:41.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:44:41.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182434 2021-08-21T14:44:41.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:41.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182434 2021-08-21T14:44:41.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2021-08-21T14:44:41.626 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182434 2021-08-21T14:44:41.627 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:44:42.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182435 -lt 107374182434 2021-08-21T14:44:42.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:44:42.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:44:42.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:44:42.075 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:44:42.075 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:44:42.075 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:44:42.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:44:42.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:44:42.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:44:42.168 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:44:42.169 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:44:42.169 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:42.169 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:42.170 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:42.170 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:44:42.170 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:42.170 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:44:42.170 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:44:42.171 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:42.171 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:42.171 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:42.171 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:42.172 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:44:42.172 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:42.172 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:42.172 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:42.173 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:44:42.173 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:42.173 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:44:42.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph osd out osd.0 2021-08-21T14:44:42.720 INFO:tasks.workunit.client.0.smithi077.stderr:osd.0 is already out. 2021-08-21T14:44:42.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:181: TEST_backfill_priority: sleep 2 2021-08-21T14:44:44.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:182: TEST_backfill_priority: flush_pg_stats 2021-08-21T14:44:44.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:44:44.732 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:44:45.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:44:45.183 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:44:45.184 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:44:45.184 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:44:45.184 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:44:45.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:44:45.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:45.185 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:44:45.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836526 2021-08-21T14:44:45.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836526 2021-08-21T14:44:45.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526' 2021-08-21T14:44:45.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:45.321 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:44:45.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673003 2021-08-21T14:44:45.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673003 2021-08-21T14:44:45.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673003' 2021-08-21T14:44:45.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:45.456 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:44:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509481 2021-08-21T14:44:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509481 2021-08-21T14:44:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673003 2-64424509481' 2021-08-21T14:44:45.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:45.592 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:44:45.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345959 2021-08-21T14:44:45.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345959 2021-08-21T14:44:45.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673003 2-64424509481 3-85899345959' 2021-08-21T14:44:45.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:45.729 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:44:45.862 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182438 2021-08-21T14:44:45.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182438 2021-08-21T14:44:45.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673003 2-64424509481 3-85899345959 4-107374182438' 2021-08-21T14:44:45.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:45.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836526 2021-08-21T14:44:45.864 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:45.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:44:45.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836526 2021-08-21T14:44:45.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:45.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836526 2021-08-21T14:44:45.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2021-08-21T14:44:45.866 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836526 2021-08-21T14:44:45.866 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:44:46.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836526 -lt 21474836526 2021-08-21T14:44:46.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:46.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673003 2021-08-21T14:44:46.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:46.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:44:46.281 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673003 2021-08-21T14:44:46.281 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:46.281 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673003 2021-08-21T14:44:46.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2021-08-21T14:44:46.283 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673003 2021-08-21T14:44:46.283 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:44:46.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673003 -lt 42949673003 2021-08-21T14:44:46.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:46.704 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509481 2021-08-21T14:44:46.704 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:46.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:44:46.705 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509481 2021-08-21T14:44:46.706 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:46.706 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509481 2021-08-21T14:44:46.707 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2021-08-21T14:44:46.707 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509481 2021-08-21T14:44:46.707 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:44:47.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509481 -lt 64424509481 2021-08-21T14:44:47.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:47.114 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345959 2021-08-21T14:44:47.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:47.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:44:47.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345959 2021-08-21T14:44:47.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:47.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345959 2021-08-21T14:44:47.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2021-08-21T14:44:47.117 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345959 2021-08-21T14:44:47.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:44:47.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345959 -lt 85899345959 2021-08-21T14:44:47.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:47.541 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182438 2021-08-21T14:44:47.541 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:47.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:44:47.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182438 2021-08-21T14:44:47.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:47.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182438 2021-08-21T14:44:47.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182438' 2021-08-21T14:44:47.543 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182438 2021-08-21T14:44:47.544 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:44:47.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182438 -lt 107374182438 2021-08-21T14:44:47.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:44:47.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:44:47.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:44:47.966 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:44:47.966 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:44:47.966 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:44:47.967 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:44:47.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:44:47.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:44:48.060 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:44:48.061 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:44:48.061 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:48.061 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:48.061 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:44:48.062 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:48.062 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:44:48.062 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:44:48.062 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:48.063 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:44:48.063 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:44:48.063 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:48.063 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:48.063 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:48.064 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:48.064 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:44:48.064 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:44:48.064 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:48.065 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:44:48.065 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:44:48.065 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:48.065 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:48.065 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:48.066 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:48.066 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:44:48.066 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:48.066 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:48.066 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:48.067 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:44:48.067 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:48.067 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:44:48.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:184: TEST_backfill_priority: ceph pg dump pgs 2021-08-21T14:44:48.436 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:44:48.436 INFO:tasks.workunit.client.0.smithi077.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2021-08-21T14:44:38.709415+0000 93'150 104:213 [1,4] 1 [1] 1 0'0 2021-08-21T14:42:42.796484+0000 0'0 2021-08-21T14:42:42.796484+0000 0 2021-08-21T14:44:48.437 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2021-08-21T14:44:45.396856+0000 93'150 105:225 [1,2] 1 [1,0] 1 0'0 2021-08-21T14:42:26.190001+0000 0'0 2021-08-21T14:42:26.190001+0000 0 2021-08-21T14:44:48.437 INFO:tasks.workunit.client.0.smithi077.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:43:14.001473+0000 93'150 104:220 [1] 1 [1] 1 0'0 2021-08-21T14:42:31.714089+0000 0'0 2021-08-21T14:42:31.714089+0000 0 2021-08-21T14:44:48.437 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:44:48.437 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:44:48.438 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:44:48.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph osd pool set test3 size 2 2021-08-21T14:44:49.126 INFO:tasks.workunit.client.0.smithi077.stderr:set pool 3 size to 2 2021-08-21T14:44:49.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:187: TEST_backfill_priority: sleep 2 2021-08-21T14:44:51.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:188: TEST_backfill_priority: flush_pg_stats 2021-08-21T14:44:51.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:44:51.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:44:51.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:44:51.522 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:44:51.522 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:44:51.523 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:44:51.523 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:44:51.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:44:51.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:51.524 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:44:51.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836529 2021-08-21T14:44:51.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836529 2021-08-21T14:44:51.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836529' 2021-08-21T14:44:51.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:51.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:44:51.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673006 2021-08-21T14:44:51.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673006 2021-08-21T14:44:51.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836529 1-42949673006' 2021-08-21T14:44:51.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:51.895 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:44:51.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509484 2021-08-21T14:44:51.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509484 2021-08-21T14:44:51.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484' 2021-08-21T14:44:51.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:51.981 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:44:52.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345962 2021-08-21T14:44:52.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345962 2021-08-21T14:44:52.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484 3-85899345962' 2021-08-21T14:44:52.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:52.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:44:52.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182441 2021-08-21T14:44:52.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182441 2021-08-21T14:44:52.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836529 1-42949673006 2-64424509484 3-85899345962 4-107374182441' 2021-08-21T14:44:52.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:52.358 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836529 2021-08-21T14:44:52.359 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:52.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:44:52.360 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836529 2021-08-21T14:44:52.360 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:52.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836529 2021-08-21T14:44:52.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2021-08-21T14:44:52.361 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836529 2021-08-21T14:44:52.361 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:44:52.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836529 -lt 21474836529 2021-08-21T14:44:52.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:52.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673006 2021-08-21T14:44:52.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:52.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:44:52.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673006 2021-08-21T14:44:52.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:52.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673006 2021-08-21T14:44:52.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2021-08-21T14:44:52.737 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673006 2021-08-21T14:44:52.737 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:44:53.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673006 -lt 42949673006 2021-08-21T14:44:53.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:53.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509484 2021-08-21T14:44:53.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:53.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:44:53.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509484 2021-08-21T14:44:53.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:53.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509484 2021-08-21T14:44:53.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2021-08-21T14:44:53.115 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509484 2021-08-21T14:44:53.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:44:53.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509482 -lt 64424509484 2021-08-21T14:44:53.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:44:54.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:44:54.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:44:54.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509485 -lt 64424509484 2021-08-21T14:44:54.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:54.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345962 2021-08-21T14:44:54.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:54.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:44:54.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345962 2021-08-21T14:44:54.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:54.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345962 2021-08-21T14:44:54.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345962' 2021-08-21T14:44:54.893 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345962 2021-08-21T14:44:54.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:44:55.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345962 -lt 85899345962 2021-08-21T14:44:55.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:55.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182441 2021-08-21T14:44:55.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:55.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:44:55.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182441 2021-08-21T14:44:55.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:55.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182441 2021-08-21T14:44:55.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182441' 2021-08-21T14:44:55.298 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182441 2021-08-21T14:44:55.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:44:55.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182441 -lt 107374182441 2021-08-21T14:44:55.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:44:55.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:44:55.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:44:55.734 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:44:55.734 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:44:55.734 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:44:55.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:44:55.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:44:55.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:44:55.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:44:55.837 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:44:55.837 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:44:55.837 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:55.838 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:55.838 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:44:55.838 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:55.838 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:44:55.838 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:44:55.839 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:55.839 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:44:55.839 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:44:55.839 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:55.840 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:55.840 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:55.840 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:55.840 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:55.841 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:44:55.841 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:44:55.841 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:55.841 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:44:55.842 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:44:55.842 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:55.842 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:55.842 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:55.843 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:55.843 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:44:55.843 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:44:55.843 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:55.843 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:44:55.844 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:44:55.844 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:55.844 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:55.844 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:55.845 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:55.845 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:44:55.845 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:55.846 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:55.846 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:55.846 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:44:55.847 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:55.847 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:44:55.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:191: TEST_backfill_priority: ceph pg dump pgs 2021-08-21T14:44:56.192 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:44:56.193 INFO:tasks.workunit.client.0.smithi077.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2021-08-21T14:44:38.709415+0000 93'150 108:216 [1,4] 1 [1] 1 0'0 2021-08-21T14:42:42.796484+0000 0'0 2021-08-21T14:42:42.796484+0000 0 2021-08-21T14:44:56.193 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2021-08-21T14:44:45.396856+0000 93'150 108:228 [1,2] 1 [1,0] 1 0'0 2021-08-21T14:42:26.190001+0000 0'0 2021-08-21T14:42:26.190001+0000 0 2021-08-21T14:44:56.193 INFO:tasks.workunit.client.0.smithi077.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2021-08-21T14:44:51.114464+0000 93'150 109:230 [1,2] 1 [1] 1 0'0 2021-08-21T14:42:31.714089+0000 0'0 2021-08-21T14:42:31.714089+0000 0 2021-08-21T14:44:56.194 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:44:56.194 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:44:56.194 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:44:56.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:44:56.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2021-08-21T14:44:56.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: PRIO=110 2021-08-21T14:44:56.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:194: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2021-08-21T14:44:56.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:44:56.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2021-08-21T14:44:56.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: eval 'ITEM="7.0"' 2021-08-21T14:44:56.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: ITEM=7.0 2021-08-21T14:44:56.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:202: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2021-08-21T14:44:56.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:44:56.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2021-08-21T14:44:56.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: PRIO=254 2021-08-21T14:44:56.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:208: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-08-21T14:44:56.213 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: seq 1 10 2021-08-21T14:44:56.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2021-08-21T14:44:56.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: ceph pg force-backfill 3.0 2021-08-21T14:44:56.214 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2021-08-21T14:44:56.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:219: TEST_backfill_priority: break 2021-08-21T14:44:56.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:227: TEST_backfill_priority: sleep 2 2021-08-21T14:44:58.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:44:58.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:44:58.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:44:58.607 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:44:58.607 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:44:58.608 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:44:58.608 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:44:58.608 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:44:58.608 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:44:58.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:229: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:44:58.721 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:44:58.721 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:44:58.721 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:58.722 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:58.722 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:44:58.722 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:58.722 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:44:58.723 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:44:58.723 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:58.723 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:44:58.723 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:44:58.724 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:58.724 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:58.724 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:58.724 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:58.725 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:58.725 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 254, 2021-08-21T14:44:58.725 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:44:58.725 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:58.725 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:44:58.726 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:44:58.726 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:58.726 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:58.726 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:58.726 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:58.727 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:44:58.727 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:44:58.727 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:44:58.727 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:44:58.728 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:44:58.728 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:44:58.728 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:58.728 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:44:58.728 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:44:58.729 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:44:58.729 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:44:58.729 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:44:58.729 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:44:58.729 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:44:58.730 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:44:58.730 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:44:58.731 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:44:58.731 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-08-21T14:44:58.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: PRIO=254 2021-08-21T14:44:58.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:231: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-08-21T14:44:58.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:236: TEST_backfill_priority: flush_pg_stats 2021-08-21T14:44:58.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:44:58.732 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:44:59.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:44:59.129 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:44:59.129 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:44:59.129 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:44:59.129 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:44:59.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:44:59.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:59.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:44:59.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836533 2021-08-21T14:44:59.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836533 2021-08-21T14:44:59.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836533' 2021-08-21T14:44:59.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:59.374 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:44:59.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673010 2021-08-21T14:44:59.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673010 2021-08-21T14:44:59.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836533 1-42949673010' 2021-08-21T14:44:59.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:59.509 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:44:59.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509488 2021-08-21T14:44:59.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509488 2021-08-21T14:44:59.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509488' 2021-08-21T14:44:59.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:59.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:44:59.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345965 2021-08-21T14:44:59.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345965 2021-08-21T14:44:59.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509488 3-85899345965' 2021-08-21T14:44:59.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:44:59.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:44:59.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182444 2021-08-21T14:44:59.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182444 2021-08-21T14:44:59.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836533 1-42949673010 2-64424509488 3-85899345965 4-107374182444' 2021-08-21T14:44:59.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:44:59.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836533 2021-08-21T14:44:59.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:44:59.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:44:59.916 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836533 2021-08-21T14:44:59.917 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:44:59.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836533 2021-08-21T14:44:59.917 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836533 2021-08-21T14:44:59.918 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2021-08-21T14:44:59.918 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:45:00.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836533 -lt 21474836533 2021-08-21T14:45:00.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:00.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673010 2021-08-21T14:45:00.293 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:00.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:45:00.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673010 2021-08-21T14:45:00.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:00.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673010 2021-08-21T14:45:00.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2021-08-21T14:45:00.295 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673010 2021-08-21T14:45:00.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:45:00.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673010 -lt 42949673010 2021-08-21T14:45:00.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:00.686 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509488 2021-08-21T14:45:00.686 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:00.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:45:00.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509488 2021-08-21T14:45:00.688 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:00.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509488 2021-08-21T14:45:00.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509488' 2021-08-21T14:45:00.689 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509488 2021-08-21T14:45:00.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:45:01.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509488 -lt 64424509488 2021-08-21T14:45:01.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:01.150 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345965 2021-08-21T14:45:01.151 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:01.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:45:01.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345965 2021-08-21T14:45:01.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:01.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345965 2021-08-21T14:45:01.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2021-08-21T14:45:01.153 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345965 2021-08-21T14:45:01.154 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:45:01.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345966 -lt 85899345965 2021-08-21T14:45:01.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:01.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182444 2021-08-21T14:45:01.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:01.563 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:45:01.563 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182444 2021-08-21T14:45:01.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:01.564 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182444 2021-08-21T14:45:01.564 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2021-08-21T14:45:01.565 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182444 2021-08-21T14:45:01.565 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:45:01.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182442 -lt 107374182444 2021-08-21T14:45:01.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:45:02.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:45:02.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:45:03.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182445 -lt 107374182444 2021-08-21T14:45:03.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:239: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2021-08-21T14:45:03.737 INFO:tasks.workunit.client.0.smithi077.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2021-08-21T14:45:03.738 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:45:03.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:240: TEST_backfill_priority: sleep 2 2021-08-21T14:45:05.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:45:05.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:45:05.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:45:05.750 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:45:05.750 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:45:05.750 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:45:05.751 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:45:05.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:45:05.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:05.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:242: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:05.865 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:05.865 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:05.865 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:05.865 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:05.865 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:05.866 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:05.866 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:05.866 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:05.866 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:05.867 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:05.867 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:05.867 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:05.867 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:05.867 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:05.868 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:05.868 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:05.868 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:05.868 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:05.869 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:05.869 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:05.869 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:05.869 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:05.869 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:05.870 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:05.870 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:05.870 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:05.870 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:05.871 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:05.871 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:05.871 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:05.871 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:05.871 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:05.872 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:05.872 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:05.872 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:05.872 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:05.872 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:05.873 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:05.873 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:05.873 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:05.873 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:05.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:05.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2021-08-21T14:45:05.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: PRIO=151 2021-08-21T14:45:05.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:244: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2021-08-21T14:45:05.876 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:05.876 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2021-08-21T14:45:05.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: eval 'ITEM="3.0"' 2021-08-21T14:45:05.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: ITEM=3.0 2021-08-21T14:45:05.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:251: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2021-08-21T14:45:05.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:05.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2021-08-21T14:45:05.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: PRIO=254 2021-08-21T14:45:05.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:257: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-08-21T14:45:05.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:264: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2021-08-21T14:45:06.233 INFO:tasks.workunit.client.0.smithi077.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2021-08-21T14:45:06.233 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:45:06.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:265: TEST_backfill_priority: sleep 5 2021-08-21T14:45:11.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:45:11.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:45:11.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:45:11.245 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:45:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:45:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:45:11.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:45:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:45:11.247 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:11.350 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:11.350 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:11.350 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:11.351 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:11.351 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:11.351 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:11.351 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:11.352 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:11.352 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:11.352 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:11.352 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:11.353 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:11.353 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:11.353 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:11.353 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:11.353 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:11.354 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:11.354 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:11.354 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:11.354 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:11.355 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:11.355 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:11.355 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:11.355 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:11.356 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:11.356 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:11.356 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:11.356 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:11.356 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:11.357 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:11.357 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:11.357 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:11.357 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:11.358 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:11.358 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:11.358 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:11.358 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:11.359 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:11.359 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:11.359 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:11.359 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:11.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:269: TEST_backfill_priority: flush_pg_stats 2021-08-21T14:45:11.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:45:11.361 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:45:11.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:45:11.775 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:45:11.776 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:45:11.776 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:45:11.776 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:45:11.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:45:11.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:11.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:45:11.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836537 2021-08-21T14:45:11.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836537 2021-08-21T14:45:11.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836537' 2021-08-21T14:45:11.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:11.913 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:45:12.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673014 2021-08-21T14:45:12.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673014 2021-08-21T14:45:12.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836537 1-42949673014' 2021-08-21T14:45:12.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:12.050 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:45:12.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509492 2021-08-21T14:45:12.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509492 2021-08-21T14:45:12.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836537 1-42949673014 2-64424509492' 2021-08-21T14:45:12.186 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:12.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:45:12.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345970 2021-08-21T14:45:12.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345970 2021-08-21T14:45:12.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836537 1-42949673014 2-64424509492 3-85899345970' 2021-08-21T14:45:12.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:12.322 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:45:12.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182449 2021-08-21T14:45:12.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182449 2021-08-21T14:45:12.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836537 1-42949673014 2-64424509492 3-85899345970 4-107374182449' 2021-08-21T14:45:12.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:12.459 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836537 2021-08-21T14:45:12.460 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:12.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:45:12.461 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836537 2021-08-21T14:45:12.461 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:12.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836537 2021-08-21T14:45:12.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2021-08-21T14:45:12.462 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836537 2021-08-21T14:45:12.462 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:45:12.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836535 -lt 21474836537 2021-08-21T14:45:12.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:45:13.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:45:13.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:45:14.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836538 -lt 21474836537 2021-08-21T14:45:14.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:14.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673014 2021-08-21T14:45:14.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:14.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:45:14.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673014 2021-08-21T14:45:14.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:14.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673014 2021-08-21T14:45:14.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673014' 2021-08-21T14:45:14.238 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673014 2021-08-21T14:45:14.239 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:45:14.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673015 -lt 42949673014 2021-08-21T14:45:14.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:14.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509492 2021-08-21T14:45:14.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:14.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:45:14.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509492 2021-08-21T14:45:14.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:14.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509492 2021-08-21T14:45:14.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2021-08-21T14:45:14.650 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509492 2021-08-21T14:45:14.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:45:15.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509493 -lt 64424509492 2021-08-21T14:45:15.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:15.057 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345970 2021-08-21T14:45:15.057 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:15.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:45:15.058 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345970 2021-08-21T14:45:15.058 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:15.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345970 2021-08-21T14:45:15.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345970' 2021-08-21T14:45:15.060 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345970 2021-08-21T14:45:15.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:45:15.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345970 -lt 85899345970 2021-08-21T14:45:15.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:15.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182449 2021-08-21T14:45:15.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:15.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:45:15.435 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182449 2021-08-21T14:45:15.436 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:15.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182449 2021-08-21T14:45:15.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182449' 2021-08-21T14:45:15.437 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182449 2021-08-21T14:45:15.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:45:15.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182449 -lt 107374182449 2021-08-21T14:45:15.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph pg force-backfill 7.0 2021-08-21T14:45:16.231 INFO:tasks.workunit.client.0.smithi077.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2021-08-21T14:45:16.232 INFO:tasks.workunit.client.0.smithi077.stderr: 2021-08-21T14:45:16.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:271: TEST_backfill_priority: sleep 2 2021-08-21T14:45:18.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:45:18.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:45:18.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:45:18.244 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:45:18.244 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:45:18.245 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:45:18.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:45:18.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:45:18.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:18.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:274: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:18.359 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:18.359 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:18.360 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:18.360 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:18.360 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:18.360 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:18.361 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:18.361 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:18.361 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:18.361 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:18.361 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:18.362 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:18.362 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:18.362 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:18.362 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:18.363 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:18.363 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:18.363 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:18.363 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:18.364 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:18.364 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:18.364 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:18.364 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:18.364 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:18.365 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:18.365 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:18.365 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:18.365 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:18.366 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:18.366 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:18.366 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:18.366 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:18.366 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:18.367 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:18.367 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:18.367 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:18.367 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:18.368 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:18.368 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:18.368 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:18.368 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:18.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:18.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2021-08-21T14:45:18.370 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: PRIO=151 2021-08-21T14:45:18.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:276: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2021-08-21T14:45:18.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:18.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2021-08-21T14:45:18.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: eval 'ITEM="7.0"' 2021-08-21T14:45:18.372 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: ITEM=7.0 2021-08-21T14:45:18.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:283: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2021-08-21T14:45:18.372 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: cat td/osd-backfill-prio/out 2021-08-21T14:45:18.372 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2021-08-21T14:45:18.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: PRIO=254 2021-08-21T14:45:18.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:289: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2021-08-21T14:45:18.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:296: TEST_backfill_priority: ceph osd unset noout 2021-08-21T14:45:19.046 INFO:tasks.workunit.client.0.smithi077.stderr:noout is unset 2021-08-21T14:45:19.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:297: TEST_backfill_priority: ceph osd unset nobackfill 2021-08-21T14:45:19.877 INFO:tasks.workunit.client.0.smithi077.stderr:nobackfill is unset 2021-08-21T14:45:19.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:299: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:45:19.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:45:19.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:45:19.890 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:45:19.890 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:45:19.890 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:45:19.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:45:19.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:299: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations' 2021-08-21T14:45:19.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations' 2021-08-21T14:45:19.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:45:19.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:45:19.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:45:19.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:45:19.892 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:45:19.892 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:45:19.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:45:19.893 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:45:19.893 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:45:19.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:45:19.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:45:19.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:45:19.979 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:45:19.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:45:20.370 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:45:20.370 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:45:20.370 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:45:20.371 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:45:20.371 INFO:tasks.workunit.client.0.smithi077.stderr:4' 2021-08-21T14:45:20.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:45:20.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:20.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:45:20.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836541 2021-08-21T14:45:20.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836541 2021-08-21T14:45:20.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836541' 2021-08-21T14:45:20.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:20.592 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:45:20.726 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673018 2021-08-21T14:45:20.727 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673018 2021-08-21T14:45:20.727 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836541 1-42949673018' 2021-08-21T14:45:20.727 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:20.728 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:45:20.862 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509496 2021-08-21T14:45:20.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509496 2021-08-21T14:45:20.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509496' 2021-08-21T14:45:20.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:20.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:45:20.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345974 2021-08-21T14:45:21.000 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345974 2021-08-21T14:45:21.000 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509496 3-85899345974' 2021-08-21T14:45:21.000 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:45:21.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:45:21.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182453 2021-08-21T14:45:21.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182453 2021-08-21T14:45:21.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836541 1-42949673018 2-64424509496 3-85899345974 4-107374182453' 2021-08-21T14:45:21.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:21.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836541 2021-08-21T14:45:21.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:21.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:45:21.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836541 2021-08-21T14:45:21.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:21.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836541 2021-08-21T14:45:21.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836541' 2021-08-21T14:45:21.139 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836541 2021-08-21T14:45:21.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:45:21.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836539 -lt 21474836541 2021-08-21T14:45:21.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:45:22.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:45:22.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:45:22.990 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836541 -lt 21474836541 2021-08-21T14:45:22.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:22.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673018 2021-08-21T14:45:22.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:22.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:45:22.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673018 2021-08-21T14:45:22.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:22.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673018 2021-08-21T14:45:22.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673018' 2021-08-21T14:45:22.994 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673018 2021-08-21T14:45:22.994 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:45:23.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673018 -lt 42949673018 2021-08-21T14:45:23.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:23.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509496 2021-08-21T14:45:23.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:23.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:45:23.485 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509496 2021-08-21T14:45:23.486 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:23.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509496 2021-08-21T14:45:23.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509496' 2021-08-21T14:45:23.488 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509496 2021-08-21T14:45:23.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:45:23.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509496 -lt 64424509496 2021-08-21T14:45:23.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:23.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345974 2021-08-21T14:45:23.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:23.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:45:23.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345974 2021-08-21T14:45:23.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:23.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345974 2021-08-21T14:45:23.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345974' 2021-08-21T14:45:23.907 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345974 2021-08-21T14:45:23.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:45:24.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345974 -lt 85899345974 2021-08-21T14:45:24.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:45:24.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182453 2021-08-21T14:45:24.435 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:45:24.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:45:24.436 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182453 2021-08-21T14:45:24.436 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:45:24.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182453 2021-08-21T14:45:24.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182453' 2021-08-21T14:45:24.437 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182453 2021-08-21T14:45:24.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:45:24.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182453 -lt 107374182453 2021-08-21T14:45:24.902 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:45:24.903 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:24.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:25.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 3 == 0 2021-08-21T14:45:25.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:25.473 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:25.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:25.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:25.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:25.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:25.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:25.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:25.888 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:25.888 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:25.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:26.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:26.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' -1 2021-08-21T14:45:26.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: wait_for_clean: loop=0 2021-08-21T14:45:26.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: num_active_clean=0 2021-08-21T14:45:26.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:26.441 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:26.441 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:26.534 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:26.534 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:26.535 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:26.535 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:26.535 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:26.535 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:26.536 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:26.536 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:26.536 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:26.536 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:26.536 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:26.537 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:26.537 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:26.537 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:26.537 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:26.538 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:26.538 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:26.538 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:26.538 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:26.539 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:26.539 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:26.539 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:26.539 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:26.539 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:26.540 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:26.540 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:26.540 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:26.540 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:26.541 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:26.541 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:26.541 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:26.541 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:26.542 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:26.542 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:26.542 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:26.542 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:26.542 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:26.543 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:26.543 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:26.543 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:26.543 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:26.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:26.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:26.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:26.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:26.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:26.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:26.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:26.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:26.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:27.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:27.070 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:27.070 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:27.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:27.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:27.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:27.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:27.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:27.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:27.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:27.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:27.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:27.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:28.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17822239 2021-08-21T14:45:28.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17822239 '!=' null 2021-08-21T14:45:28.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:28.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:28.120 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:28.121 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:28.214 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:28.215 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:28.215 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:28.215 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:28.215 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:28.216 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:28.216 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:28.216 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:28.216 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:28.217 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:28.217 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:28.217 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:28.217 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:28.217 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:28.218 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:28.218 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:28.218 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:28.218 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:28.219 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:28.219 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:28.219 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:28.219 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:28.219 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:28.220 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:28.220 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:28.220 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:28.220 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:28.221 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:28.221 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:28.221 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:28.221 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:28.222 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:28.222 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:28.222 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:28.222 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:28.222 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:28.223 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:28.223 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:28.223 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:28.223 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:28.224 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:28.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:28.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:28.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:28.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:28.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:28.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:28.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:28.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:28.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:28.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:28.743 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:28.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:28.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:29.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:29.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:29.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:29.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:29.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:29.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:29.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:29.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:29.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:29.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17822239 2021-08-21T14:45:29.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17822239 '!=' null 2021-08-21T14:45:29.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:29.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:29.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:29.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:29.966 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:29.966 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:29.966 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:29.966 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:29.967 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:29.967 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:29.967 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:29.967 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:29.968 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:29.968 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:29.968 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:29.968 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:29.968 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:29.969 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:29.969 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:29.969 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:29.969 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:29.970 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:29.970 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:29.970 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:29.970 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:29.970 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:29.971 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:29.971 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:29.971 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:29.971 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:29.972 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:29.972 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:29.972 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:29.972 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:29.972 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:29.973 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:29.973 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:29.973 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:29.973 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:29.974 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:29.974 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:29.974 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:29.974 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:29.974 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:29.975 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:29.975 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:30.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:30.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:30.077 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:30.077 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:30.077 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:30.077 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:30.078 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:30.078 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:30.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:30.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:30.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:30.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:30.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:30.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:30.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:30.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:30.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:30.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:30.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:30.944 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:30.944 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:31.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=14677677 2021-08-21T14:45:31.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 14677677 '!=' null 2021-08-21T14:45:31.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:31.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:31.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:31.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:31.538 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:31.539 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:31.539 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:31.539 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:31.539 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:31.540 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:31.540 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:31.540 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:31.540 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:31.540 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:31.541 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:31.541 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:31.541 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:31.541 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:31.542 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:31.542 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:31.542 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:31.542 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:31.542 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:31.543 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:31.543 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:31.543 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:31.543 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:31.543 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:31.544 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:31.544 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:31.544 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:31.544 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:31.545 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:31.545 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:31.545 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:31.545 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:31.545 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:31.546 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:31.546 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:31.546 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:31.546 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:31.546 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:31.547 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:31.547 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:31.547 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:31.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:31.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:31.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:31.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:31.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:31.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:32.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:32.021 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:32.021 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:32.021 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:32.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:32.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:32.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:32.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:32.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:32.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:32.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:32.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:32.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:33.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=12275936 2021-08-21T14:45:33.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 12275936 '!=' null 2021-08-21T14:45:33.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:33.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:33.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:33.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:33.207 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:33.207 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:33.207 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:33.208 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:33.208 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:33.208 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:33.208 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:33.208 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:33.209 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:33.209 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:33.209 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:33.209 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:33.209 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:33.210 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:33.210 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:33.210 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:33.211 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:33.211 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:33.211 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:33.211 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:33.212 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:33.212 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:33.212 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:33.212 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:33.212 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:33.213 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:33.213 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:33.213 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:33.213 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:33.213 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:33.214 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:33.214 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:33.214 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:33.214 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:33.215 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:33.215 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:33.215 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:33.216 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:33.216 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:33.216 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:33.216 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:33.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:33.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:33.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:33.318 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:33.318 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:33.318 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:33.319 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:33.319 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:33.319 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:33.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:33.695 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:33.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:33.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:34.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:34.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:34.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:34.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:34.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:34.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:34.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:34.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:34.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:34.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=20968018 2021-08-21T14:45:34.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 20968018 '!=' null 2021-08-21T14:45:34.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:34.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:34.836 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:34.836 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:34.929 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:34.930 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:34.930 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:34.930 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:34.930 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:34.931 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:34.931 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:34.931 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:34.931 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:34.932 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:34.932 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:34.932 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:34.932 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:34.932 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:34.933 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:34.933 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:34.933 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:34.933 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:34.933 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:34.934 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:34.934 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:34.934 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:34.934 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:34.935 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:34.935 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:34.935 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:34.935 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:34.935 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:34.936 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:34.936 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:34.936 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:34.936 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:34.936 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:34.937 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:34.937 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:34.937 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:34.937 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:34.937 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:34.938 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:34.938 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:34.938 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:34.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:35.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:35.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:35.040 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:35.040 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:35.040 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:35.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:35.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:35.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:35.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:35.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:35.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:35.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:36.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:36.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:36.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:36.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:36.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:36.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:36.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:36.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:36.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:36.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17473339 2021-08-21T14:45:36.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17473339 '!=' null 2021-08-21T14:45:36.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:36.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:36.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:36.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:36.652 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:36.653 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:36.653 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:36.653 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:36.653 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:36.654 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:36.654 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:36.654 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:36.654 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:36.655 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:36.655 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:36.655 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:36.655 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:36.656 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:36.656 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:36.656 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:36.656 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:36.656 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:36.657 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:36.657 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:36.657 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:36.657 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:36.658 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:36.658 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:36.658 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:36.658 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:36.659 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:36.659 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:36.659 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:36.659 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:36.659 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:36.660 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:36.660 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:36.660 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:36.660 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:36.661 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:36.661 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:36.661 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:36.661 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:36.661 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:36.662 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:36.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:36.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:36.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:36.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:36.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:36.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:36.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:36.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:36.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:37.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:37.266 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:37.266 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:37.267 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:37.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:37.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:37.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:37.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:37.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:37.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:37.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:37.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:37.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:38.258 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=25336473 2021-08-21T14:45:38.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 25336473 '!=' null 2021-08-21T14:45:38.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:38.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:38.259 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:38.260 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:38.355 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:38.364 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:38.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:38.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:38.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:38.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:38.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:38.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:38.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:38.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:38.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:38.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:38.834 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:38.835 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:38.835 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:39.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:39.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:39.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:39.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:39.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:39.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:39.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:39.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:39.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:39.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=25336473 2021-08-21T14:45:39.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 25336473 '!=' null 2021-08-21T14:45:39.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:39.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:39.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:39.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:39.985 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:39.985 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:39.986 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:39.986 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:39.986 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:39.986 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:39.987 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:39.987 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:39.987 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:39.987 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:39.987 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:39.988 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:39.988 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:39.988 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:39.988 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:39.989 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:39.989 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:39.989 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:39.989 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:39.989 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:39.990 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:39.990 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:39.990 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:39.990 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:39.991 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:39.991 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:39.991 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:39.991 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:39.991 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:39.992 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:39.992 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:39.992 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:39.992 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:39.993 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:39.993 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:39.993 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:39.993 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:39.993 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:39.994 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:39.994 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:39.994 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:39.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:40.096 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:40.096 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:40.096 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:40.096 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:40.097 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:40.097 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:40.097 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:40.097 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:40.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:40.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:40.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:40.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:41.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:41.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:41.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:41.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:41.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:41.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:41.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:41.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:41.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:41.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599641 2021-08-21T14:45:41.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599641 '!=' null 2021-08-21T14:45:41.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:41.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:41.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:41.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:41.696 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:41.696 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:41.696 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:41.697 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:41.698 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:41.698 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:41.698 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:41.698 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:41.698 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:41.699 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:41.699 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:41.699 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:41.699 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:41.700 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:41.700 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:41.700 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:41.700 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:41.700 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:41.701 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:41.701 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:41.701 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:41.701 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:41.701 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:41.702 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:41.702 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:41.702 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:41.702 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:41.703 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:41.703 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:41.703 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:41.703 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:41.703 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:41.704 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:41.704 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:41.704 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:41.704 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:41.705 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:41.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:41.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:41.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:41.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:41.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:41.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:41.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:41.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:41.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:42.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:42.234 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:42.234 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:42.234 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:42.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:42.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:42.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:42.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:42.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:42.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:42.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:42.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:42.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:43.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599327 2021-08-21T14:45:43.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599327 '!=' null 2021-08-21T14:45:43.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:43.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:43.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:43.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:43.423 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:43.423 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:43.423 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:43.424 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:43.424 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:43.424 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:43.424 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:43.425 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:43.425 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:43.425 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:43.425 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:43.425 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:43.426 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:43.426 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:43.426 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:43.426 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:43.427 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:43.427 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:43.427 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:43.427 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:43.427 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:43.428 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:43.428 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:43.428 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:43.428 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:43.428 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:43.429 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:43.429 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:43.429 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:43.429 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:43.429 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:43.430 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:43.430 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:43.430 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:43.431 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:43.431 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:43.431 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:43.431 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:43.431 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:43.432 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:43.432 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:43.432 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:43.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:43.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:43.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:43.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:43.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:43.535 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:43.535 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:43.535 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:43.910 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:43.910 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:43.910 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:43.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:44.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:44.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:44.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:44.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:44.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:44.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:44.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:44.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:44.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:45.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=25335731 2021-08-21T14:45:45.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 25335731 '!=' null 2021-08-21T14:45:45.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:45.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:45.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:45.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:45.099 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:45.099 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:45.099 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:45.099 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:45.100 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:45.100 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:45.100 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:45.100 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:45.101 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:45.101 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:45.101 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:45.101 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:45.101 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:45.102 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:45.102 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:45.102 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:45.102 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 151, 2021-08-21T14:45:45.103 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:45.103 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:45.103 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:45.103 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:45.103 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:45.104 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:45.104 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:45.104 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:45.104 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:45.105 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:45.105 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:45.105 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "7.0", 2021-08-21T14:45:45.105 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 254, 2021-08-21T14:45:45.105 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:45.106 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:45.106 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:45.106 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:45.106 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:45.106 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:45.107 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:45.107 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:45.107 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:45.107 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:45.107 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:45.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:45.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:45.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:45.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:45.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:45.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:45.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:45.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:45.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:45.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:45.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:45.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:45.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:46.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:46.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:46.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:46.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:46.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:46.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:46.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:46.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:46.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:46.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599242 2021-08-21T14:45:46.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599242 '!=' null 2021-08-21T14:45:46.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:46.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:46.836 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:46.836 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:46.932 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:46.932 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:46.932 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:46.933 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:46.933 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:46.933 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:46.934 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:46.934 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:46.934 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:46.934 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:46.935 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:46.935 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:46.935 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:46.935 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:46.936 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:46.936 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:46.936 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:46.936 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:46.936 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:46.937 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:46.937 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:46.937 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:46.937 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:46.937 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:46.938 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:46.938 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:46.938 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:46.938 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:46.939 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:46.939 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:46.939 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:46.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:47.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:47.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:47.043 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:47.043 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:47.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:47.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:47.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:47.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:47.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:45:47.684 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:47.684 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:47.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:48.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 3 2021-08-21T14:45:48.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:45:48.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:48.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:48.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:48.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:48.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:48.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:48.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:48.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24898863 2021-08-21T14:45:48.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24898863 '!=' null 2021-08-21T14:45:48.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:48.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:48.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:48.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:48.888 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:48.888 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:48.889 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:48.889 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:48.889 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:48.889 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:48.889 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:48.890 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:48.890 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:48.890 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:48.890 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:48.891 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:48.891 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:48.891 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:48.891 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:48.892 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:48.892 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:48.892 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:48.892 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:48.892 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:48.893 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:48.893 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:48.893 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:48.893 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:48.894 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:48.894 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:48.894 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:48.894 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:48.894 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:48.895 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:48.895 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:48.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:48.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:48.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:49.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:49.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:49.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:49.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:49.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:49.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:49.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:49.401 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:49.402 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:49.402 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:50.016 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:50.016 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 0 2021-08-21T14:45:50.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: wait_for_clean: loop=0 2021-08-21T14:45:50.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: num_active_clean=1 2021-08-21T14:45:50.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:50.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:50.018 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:50.111 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:50.111 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:50.111 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:50.112 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:50.112 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:50.112 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:50.112 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:50.113 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:50.113 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:50.113 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:50.113 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:50.114 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:50.114 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:50.114 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:50.114 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:50.114 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:50.115 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:50.115 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:50.115 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:50.115 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:50.115 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:50.116 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:50.116 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:50.116 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:50.116 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:50.116 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:50.117 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:50.117 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:50.117 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:50.117 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:50.118 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:50.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:50.220 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:50.221 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:50.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:50.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:50.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:50.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:50.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:50.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:50.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:50.703 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:50.703 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:50.703 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:51.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:51.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:45:51.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:51.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:51.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:51.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:51.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:51.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:51.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:51.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24899463 2021-08-21T14:45:51.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24899463 '!=' null 2021-08-21T14:45:51.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:51.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:51.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:51.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:52.006 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:52.007 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:52.007 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:52.007 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:52.007 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:52.007 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:52.008 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:52.008 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:52.008 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:52.008 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:52.008 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:52.009 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:52.009 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:52.009 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:52.009 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:52.009 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:52.010 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:52.010 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:52.010 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:52.010 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:52.010 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:52.011 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:52.011 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:52.011 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:52.011 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:52.011 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:52.012 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:52.012 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:52.012 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:52.012 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:52.012 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:52.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:52.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:52.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:52.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:52.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:52.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:52.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:52.119 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:52.119 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:52.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:52.487 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:52.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:52.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:52.990 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:52.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:45:52.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:52.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:52.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:52.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:52.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:52.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:52.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:53.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=11794501 2021-08-21T14:45:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 11794501 '!=' null 2021-08-21T14:45:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:53.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:53.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:53.734 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:53.734 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:53.735 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:53.735 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:53.735 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:53.735 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:53.735 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:53.736 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:53.736 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:53.736 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:53.736 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:53.737 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:53.737 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:53.737 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:53.737 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:53.737 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:53.738 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:53.738 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:53.738 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:53.738 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:53.739 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:53.739 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:53.739 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:53.739 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:53.740 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:53.740 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:53.740 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:53.740 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:53.741 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:53.741 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:53.741 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:53.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:53.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:53.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:53.845 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:53.845 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:53.845 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:53.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:53.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:53.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:54.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:54.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:54.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:54.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:54.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:54.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:45:54.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:54.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:54.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:54.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:54.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:54.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:54.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:55.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=23588640 2021-08-21T14:45:55.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 23588640 '!=' null 2021-08-21T14:45:55.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:55.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:55.314 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:55.314 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:55.407 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:55.408 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:55.408 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:55.408 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:55.408 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:55.409 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:55.409 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:55.409 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:55.409 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:55.409 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:55.410 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:55.410 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:55.410 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:55.410 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:55.411 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:55.411 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:55.411 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:55.411 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:55.412 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:55.412 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:55.412 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:55.412 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:55.412 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:55.413 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:55.413 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:55.413 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:55.413 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:55.414 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:55.414 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:55.414 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:55.414 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:55.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:55.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:55.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:55.518 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:55.518 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:55.519 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:55.519 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:55.519 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:55.519 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:55.907 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:55.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:55.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:55.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:56.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:56.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:45:56.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:56.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:56.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:56.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:56.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:56.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:56.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:56.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=19487575 2021-08-21T14:45:56.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 19487575 '!=' null 2021-08-21T14:45:56.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:56.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:56.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:56.932 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:57.027 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:57.028 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:57.028 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:57.028 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:57.028 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:57.029 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:57.029 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:57.029 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:57.029 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:57.030 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:57.030 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:57.030 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:57.030 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:57.031 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:57.031 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:57.031 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:57.031 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:57.031 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:57.032 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:57.032 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:57.032 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:57.032 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:57.033 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:57.033 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:57.033 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:57.033 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:57.034 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:57.034 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:57.034 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:57.034 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:57.034 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:57.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:57.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:57.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:57.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:57.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:57.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:57.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:57.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:57.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:57.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:57.512 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:57.512 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:57.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:58.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:58.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:45:58.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:58.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:58.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:58.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:58.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:58.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:58.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:45:58.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=27147427 2021-08-21T14:45:58.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 27147427 '!=' null 2021-08-21T14:45:58.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:45:58.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:58.702 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:45:58.703 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:45:58.796 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:45:58.796 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:45:58.796 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:58.797 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:58.797 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:45:58.797 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:58.797 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:45:58.798 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:45:58.798 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:58.798 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:45:58.798 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:45:58.798 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:58.799 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:58.799 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:58.799 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:58.799 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:45:58.800 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:45:58.800 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:45:58.800 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:45:58.800 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:45:58.800 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:45:58.801 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:58.801 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:45:58.801 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:45:58.801 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:45:58.802 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:45:58.802 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:45:58.802 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:45:58.802 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:45:58.802 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:45:58.803 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:45:58.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:45:58.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:45:58.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:45:58.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:45:58.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:45:58.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:45:58.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:45:58.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:45:58.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:45:59.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:45:59.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:45:59.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:45:59.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:45:59.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:45:59.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:45:59.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:45:59.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:45:59.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:45:59.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:45:59.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:45:59.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:45:59.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:00.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599408 2021-08-21T14:46:00.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599408 '!=' null 2021-08-21T14:46:00.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:00.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:00.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:00.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:00.470 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:00.470 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:00.470 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:00.471 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:00.471 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:00.471 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:00.471 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:00.472 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:00.472 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:00.472 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:00.472 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:00.472 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:00.473 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:00.473 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:00.473 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:00.473 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:00.474 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:00.474 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:00.474 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:00.474 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:00.475 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:00.475 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:00.475 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:00.475 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:00.475 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:00.476 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:00.476 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:00.476 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:00.476 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:00.476 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:00.477 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:00.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:00.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:00.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:00.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:00.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:00.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:00.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:00.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:00.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:00.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:00.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:00.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:00.999 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:01.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:01.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:01.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:01.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:01.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:01.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:01.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:01.571 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:01.571 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:02.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599373 2021-08-21T14:46:02.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599373 '!=' null 2021-08-21T14:46:02.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:02.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:02.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:02.238 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:02.330 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:02.331 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:02.331 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:02.331 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:02.331 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:02.332 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:02.332 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:02.332 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:02.332 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:02.333 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:02.333 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:02.333 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:02.333 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:02.333 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:02.334 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:02.334 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:02.334 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:02.334 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:02.335 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:02.335 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:02.335 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:02.335 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:02.335 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:02.336 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:02.336 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:02.336 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:02.336 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:02.337 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:02.337 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:02.337 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:02.337 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:02.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:02.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:02.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:02.441 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:02.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:02.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:02.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:02.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:02.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:02.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:02.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:02.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:02.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:03.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:03.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:03.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:03.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:03.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:03.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:03.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:03.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:03.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:04.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599373 2021-08-21T14:46:04.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599373 '!=' null 2021-08-21T14:46:04.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:04.066 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:04.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:04.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:04.160 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:04.160 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:04.161 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:04.161 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:04.161 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:04.161 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:04.162 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:04.162 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:04.162 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:04.162 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:04.162 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:04.163 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:04.163 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:04.163 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:04.163 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:04.164 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:04.164 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:04.164 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:04.164 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:04.164 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:04.165 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:04.165 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:04.165 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:04.165 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:04.166 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:04.166 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:04.166 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:04.166 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:04.166 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:04.167 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:04.167 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:04.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:04.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:04.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:04.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:04.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:04.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:04.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:04.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:04.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:04.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:04.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:04.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:04.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:05.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:05.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:05.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:05.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:05.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:05.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:05.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:05.281 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:05.281 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:05.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=25335813 2021-08-21T14:46:05.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 25335813 '!=' null 2021-08-21T14:46:05.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:05.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:05.780 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:05.780 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:05.873 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:05.873 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:05.873 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:05.874 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:05.874 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:05.874 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:05.874 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:05.875 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:05.875 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:05.875 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:05.875 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:05.875 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:05.876 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:05.876 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:05.876 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:05.876 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:05.876 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:05.877 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:05.877 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:05.877 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:05.877 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:05.878 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:05.878 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:05.878 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:05.878 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:05.878 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:05.879 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:05.879 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:05.879 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:05.879 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:05.879 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:05.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:05.983 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:05.983 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:05.983 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:05.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:05.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:05.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:05.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:05.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:06.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:06.354 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:06.354 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:06.355 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:06.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:06.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:06.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:06.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:06.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:06.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:06.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:06.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:06.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:07.438 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17473084 2021-08-21T14:46:07.438 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17473084 '!=' null 2021-08-21T14:46:07.439 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:07.439 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:07.439 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:07.439 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:07.533 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:07.533 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:07.534 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:07.534 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:07.534 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:07.534 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:07.535 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:07.535 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:07.535 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:07.535 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:07.536 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:07.536 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:07.536 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:07.536 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:07.537 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:07.537 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:07.537 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:07.538 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:07.538 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:07.538 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:07.538 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:07.539 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:07.539 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:07.539 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:07.539 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:07.540 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:07.540 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:07.540 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:07.540 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:07.540 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:07.541 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:07.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:07.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:07.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:07.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:07.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:07.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:07.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:07.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:07.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:08.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:08.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:08.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:08.018 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:08.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:08.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:08.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:08.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:08.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:08.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:08.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:08.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:08.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:09.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=25336024 2021-08-21T14:46:09.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 25336024 '!=' null 2021-08-21T14:46:09.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:09.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:09.129 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:09.130 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:09.130 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:09.130 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:09.130 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:09.131 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:09.131 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:09.131 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:09.131 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:09.131 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:09.132 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:09.132 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:09.132 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:09.132 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:09.133 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:09.133 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:09.133 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:09.133 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:09.133 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:09.134 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:09.134 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:09.134 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:09.134 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:09.134 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:09.135 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:09.135 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:09.135 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:09.135 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:09.136 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:09.136 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:09.136 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:09.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:09.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:09.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:09.240 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:09.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:09.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:09.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:09.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:09.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:09.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:09.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:09.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:09.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:10.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:10.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:10.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:10.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:10.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:10.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:10.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:10.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:10.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:10.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599437 2021-08-21T14:46:10.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599437 '!=' null 2021-08-21T14:46:10.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:10.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:10.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:10.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:10.748 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:10.748 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:10.748 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:10.749 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:10.749 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [ 2021-08-21T14:46:10.749 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:10.749 INFO:tasks.workunit.client.0.smithi077.stdout: "priority": 110, 2021-08-21T14:46:10.749 INFO:tasks.workunit.client.0.smithi077.stdout: "items": [ 2021-08-21T14:46:10.750 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:10.750 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:10.750 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:10.750 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:10.751 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:10.751 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:10.751 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:10.751 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:46:10.752 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:10.752 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:10.752 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "3.0", 2021-08-21T14:46:10.752 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 151, 2021-08-21T14:46:10.753 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:10.753 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:10.753 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:10.753 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:10.754 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:10.754 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:10.754 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:10.754 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:10.755 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:10.755 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:10.755 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:10.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:10.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:10.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:10.859 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:10.859 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:10.859 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:10.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:10.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:10.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:11.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:11.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:11.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:11.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:11.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:11.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:11.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:11.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:11.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:11.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:11.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:11.831 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:11.831 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:12.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599434 2021-08-21T14:46:12.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599434 '!=' null 2021-08-21T14:46:12.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:12.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:12.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:12.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:12.593 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:12.593 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:12.594 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:12.594 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:12.594 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:12.594 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:12.595 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:12.595 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:12.595 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:12.595 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:12.595 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:12.596 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:12.596 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:12.596 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:12.596 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:12.596 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:12.597 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:12.597 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:12.597 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:12.597 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:12.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:12.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:12.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:12.704 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:12.704 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:12.705 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:12.705 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:12.705 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:12.705 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:13.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:13.127 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:13.127 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:13.127 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:13.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:13.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:13.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:13.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:13.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:13.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:13.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:13.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:13.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:14.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24899420 2021-08-21T14:46:14.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24899420 '!=' null 2021-08-21T14:46:14.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:14.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:14.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:14.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:14.656 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:14.656 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:14.656 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:14.656 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:14.657 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:14.657 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:14.657 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:14.657 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:14.658 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:14.658 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:14.658 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:14.658 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:14.659 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:14.659 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:14.659 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:14.659 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:14.659 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:14.660 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:14.660 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:14.660 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:14.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:14.766 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:14.767 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:14.767 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:14.768 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:14.768 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:14.769 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:14.769 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:14.769 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:15.132 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:15.133 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:15.133 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:15.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:15.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:15.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:15.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:15.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:15.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:15.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:15.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:15.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:15.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:16.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=13104966 2021-08-21T14:46:16.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 13104966 '!=' null 2021-08-21T14:46:16.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:16.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:16.332 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:16.332 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:16.425 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:16.426 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:16.426 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:16.426 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:16.427 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:16.427 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:16.427 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:16.427 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:16.427 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:16.428 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:16.428 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:16.428 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:16.428 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:16.429 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:16.429 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:16.429 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:16.429 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:16.429 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:16.430 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:16.430 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:16.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:16.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:16.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:16.536 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:16.537 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:16.537 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:16.537 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:16.537 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:16.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:16.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:46:16.906 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:16.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:16.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:17.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 3 2021-08-21T14:46:17.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 1 '!=' 1 2021-08-21T14:46:17.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:17.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:17.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:17.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:17.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:17.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:17.583 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:18.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=13104966 2021-08-21T14:46:18.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 13104966 '!=' null 2021-08-21T14:46:18.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:18.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:18.158 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:18.159 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:18.251 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:18.251 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:18.252 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:18.252 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:18.252 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:18.252 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:18.253 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:18.253 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:18.253 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:18.253 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:18.254 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:18.254 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:18.254 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:18.254 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:18.254 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:18.255 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:18.255 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:18.255 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:18.255 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:18.256 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:18.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:18.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:18.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:18.362 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:18.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:18.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:18.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:18.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:18.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:18.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:18.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:18.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:18.747 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:19.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:19.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 1 2021-08-21T14:46:19.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: wait_for_clean: loop=0 2021-08-21T14:46:19.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: num_active_clean=2 2021-08-21T14:46:19.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:19.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:19.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:19.336 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:19.336 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:19.337 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:19.337 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:19.337 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:19.337 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:19.338 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:19.338 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:19.338 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:19.338 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:19.338 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:19.339 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:19.339 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:19.339 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:19.339 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:19.340 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:19.340 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:19.340 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:19.343 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:19.344 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:19.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:19.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:19.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:19.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:19.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:19.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:19.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:19.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:19.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:19.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:19.837 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:19.838 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:19.838 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:20.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:20.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:20.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:20.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:20.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:20.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:20.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:20.353 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:20.354 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:20.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24899664 2021-08-21T14:46:20.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24899664 '!=' null 2021-08-21T14:46:20.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:20.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:20.978 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:20.978 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:21.073 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:21.073 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:21.073 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:21.073 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:21.074 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:21.074 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:21.074 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:21.074 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:21.075 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:21.075 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:21.075 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:21.075 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:21.076 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:21.076 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:21.076 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:21.076 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:21.077 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:21.077 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:21.077 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:21.077 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:21.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:21.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:21.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:21.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:21.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:21.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:21.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:21.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:21.185 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:21.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:21.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:21.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:21.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:22.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:22.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:22.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:22.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:22.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:22.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:22.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:22.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:22.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:22.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=10413522 2021-08-21T14:46:22.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 10413522 '!=' null 2021-08-21T14:46:22.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:22.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:22.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:22.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:22.637 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:22.637 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:22.637 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:22.638 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:22.638 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:22.638 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:22.638 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:22.639 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:22.639 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:22.639 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:22.639 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:22.639 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:22.640 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:22.640 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:22.640 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:22.640 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:22.641 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:22.641 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:22.641 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:22.641 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:22.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:22.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:22.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:22.748 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:22.748 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:22.748 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:22.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:22.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:22.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:23.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:23.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:23.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:23.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:23.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:23.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:23.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:23.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:23.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:23.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:23.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:23.715 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:23.715 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:24.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17847820 2021-08-21T14:46:24.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17847820 '!=' null 2021-08-21T14:46:24.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:24.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:24.266 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:24.266 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:24.358 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:24.359 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:24.359 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:24.359 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:24.359 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:24.360 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:24.360 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:24.360 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:24.360 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:24.361 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:24.361 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:24.361 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:24.361 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:24.361 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:24.362 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:24.362 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:24.362 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:24.362 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:24.363 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:24.363 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:24.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:24.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:24.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:24.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:24.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:24.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:24.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:24.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:24.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:24.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:24.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:24.841 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:24.841 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:25.336 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:25.336 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:25.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:25.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:25.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:25.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:25.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:25.338 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:25.338 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:25.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17847820 2021-08-21T14:46:25.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17847820 '!=' null 2021-08-21T14:46:25.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:25.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:25.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:25.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:25.933 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:25.934 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:25.934 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:25.934 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:25.934 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:25.935 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:25.935 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:25.935 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:25.935 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:25.935 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:25.936 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:25.936 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:25.936 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:25.936 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:25.936 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:25.937 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:25.937 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:25.937 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:25.937 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:25.937 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:25.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:26.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:26.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:26.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:26.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:26.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:26.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:26.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:26.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:26.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:26.413 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:26.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:26.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:26.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:26.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:26.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:26.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:26.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:26.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:26.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:26.944 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:26.944 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:27.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=15726011 2021-08-21T14:46:27.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 15726011 '!=' null 2021-08-21T14:46:27.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:27.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:27.494 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:27.494 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:27.587 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:27.587 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:27.588 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:27.588 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:27.588 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:27.588 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:27.589 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:27.589 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:27.589 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:27.589 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:27.590 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:27.590 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:27.590 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:27.590 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:27.591 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:27.591 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:27.591 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:27.591 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:27.591 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:27.592 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:27.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:27.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:27.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:27.698 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:27.699 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:27.699 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:27.699 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:27.699 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:27.700 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:28.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:28.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:28.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:28.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:28.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:28.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:28.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:28.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:28.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:28.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:28.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:28.718 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:28.718 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:29.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24462663 2021-08-21T14:46:29.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24462663 '!=' null 2021-08-21T14:46:29.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:29.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:29.265 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:29.265 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:29.361 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:29.361 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:29.361 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:29.362 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:29.362 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:29.362 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:29.362 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:29.363 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:29.363 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:29.363 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:29.363 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:29.364 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:29.364 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:29.364 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:29.364 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:29.364 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:29.365 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:29.365 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:29.365 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:29.365 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:29.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:29.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:29.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:29.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:29.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:29.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:29.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:29.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:29.473 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:29.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:29.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:29.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:29.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:30.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:30.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:30.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:30.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:30.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:30.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:30.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:30.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:30.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:31.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599551 2021-08-21T14:46:31.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599551 '!=' null 2021-08-21T14:46:31.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:31.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:31.024 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:31.025 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:31.117 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:31.117 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:31.117 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:31.118 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:31.118 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:31.118 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:31.118 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:31.119 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:31.119 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:31.119 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:31.119 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:31.119 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:31.120 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:31.120 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:31.120 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:31.120 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:31.121 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:31.121 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:31.121 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:31.121 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:31.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:31.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:31.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:31.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:31.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:31.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:31.229 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:31.229 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:31.229 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:31.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:31.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:31.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:31.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:32.187 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:32.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:32.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:32.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:32.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:32.189 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:32.189 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:32.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:32.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:32.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599506 2021-08-21T14:46:32.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599506 '!=' null 2021-08-21T14:46:32.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:32.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:32.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:32.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:33.051 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:33.051 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:33.051 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:33.052 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:33.052 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:33.052 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:33.052 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:33.052 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:33.053 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:33.053 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:33.053 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:33.053 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:33.054 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:33.054 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:33.054 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:33.055 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:33.055 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:33.055 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:33.055 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:33.055 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:33.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:33.162 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:33.162 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:33.163 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:33.163 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:33.163 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:33.163 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:33.164 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:33.164 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:33.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:33.529 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:33.529 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:33.529 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:34.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:34.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:34.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:34.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:34.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:34.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:34.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:34.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:34.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:34.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=24462600 2021-08-21T14:46:34.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 24462600 '!=' null 2021-08-21T14:46:34.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:34.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:34.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:34.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:34.814 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:34.814 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:34.814 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:34.815 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:34.815 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:34.815 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:34.815 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:34.816 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:34.816 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:34.816 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:34.816 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:34.816 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:34.817 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:34.817 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:34.817 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:34.817 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:34.818 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:34.818 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:34.818 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:34.818 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:34.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:34.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:34.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:34.926 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:34.926 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:34.926 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:34.926 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:34.927 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:34.927 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:35.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:35.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:35.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:35.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:35.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:35.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:35.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:35.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:35.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:35.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:35.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:35.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:35.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:36.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599570 2021-08-21T14:46:36.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599570 '!=' null 2021-08-21T14:46:36.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:36.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:36.349 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:36.349 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:36.442 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:36.443 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:36.443 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:36.443 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:36.443 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:36.444 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:36.444 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:36.444 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:36.444 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:36.444 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:36.445 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:36.445 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:36.445 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:36.445 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:36.446 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:36.446 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:36.446 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:36.446 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:36.446 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:36.447 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:36.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:36.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:36.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:36.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:36.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:36.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:36.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:36.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:36.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:36.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:36.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:36.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:36.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:37.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:37.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:37.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:37.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:37.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:37.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:37.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:37.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:37.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:38.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=16599570 2021-08-21T14:46:38.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 16599570 '!=' null 2021-08-21T14:46:38.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:38.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:38.038 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:38.038 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:38.130 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:38.131 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:38.131 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:38.131 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:38.131 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:38.131 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [ 2021-08-21T14:46:38.132 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:38.132 INFO:tasks.workunit.client.0.smithi077.stdout: "item": "1.0", 2021-08-21T14:46:38.132 INFO:tasks.workunit.client.0.smithi077.stdout: "prio": 110, 2021-08-21T14:46:38.132 INFO:tasks.workunit.client.0.smithi077.stdout: "can_preempt": true 2021-08-21T14:46:38.133 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:38.133 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:38.133 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:38.133 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:38.134 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:38.134 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:38.134 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:38.134 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:38.134 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:38.135 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:38.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:38.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:38.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:38.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:38.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:38.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:38.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:38.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:38.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:38.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:38.610 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:38.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:38.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:39.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:39.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:39.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:39.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:39.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:39.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:39.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:39.741 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=25336172 2021-08-21T14:46:39.741 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 25336172 '!=' null 2021-08-21T14:46:39.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:39.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:39.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:39.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:39.835 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:39.835 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:39.835 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:39.835 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:39.836 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:39.836 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:39.836 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:39.836 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:39.837 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:39.837 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:39.837 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:39.837 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:39.838 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:39.838 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:39.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:39.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:39.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:39.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:39.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:39.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:39.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:39.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:39.947 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:40.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:40.314 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:40.314 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:40.314 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:40.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:40.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:40.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:40.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:40.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:40.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:40.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:40.885 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:40.885 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:41.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=19919567 2021-08-21T14:46:41.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 19919567 '!=' null 2021-08-21T14:46:41.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:41.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:41.402 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:41.402 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:41.495 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:41.495 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:41.495 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:41.496 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:41.496 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:41.496 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:41.496 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:41.496 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:41.497 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:41.497 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:41.497 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:41.497 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:41.498 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:41.498 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:41.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:41.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:41.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:41.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:41.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:41.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:41.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:41.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:41.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:41.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:41.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:41.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:41.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:42.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:42.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:42.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:42.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:42.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:42.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:42.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:42.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:42.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:42.989 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=13104922 2021-08-21T14:46:42.989 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 13104922 '!=' null 2021-08-21T14:46:42.990 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:42.990 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:42.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:42.991 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:43.081 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:43.081 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:43.082 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:43.082 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:43.082 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:43.082 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:43.083 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:43.083 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:43.083 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:43.083 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:43.083 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:43.084 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:43.084 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:43.084 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:43.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:43.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:43.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:43.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:43.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:43.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:43.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:43.193 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:43.193 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:43.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=2 2021-08-21T14:46:43.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:43.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:43.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:44.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 2 = 3 2021-08-21T14:46:44.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 2 '!=' 2 2021-08-21T14:46:44.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:46:44.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:46:44.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:46:44.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:46:44.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:46:44.058 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:46:44.058 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:46:44.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=17036352 2021-08-21T14:46:44.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 17036352 '!=' null 2021-08-21T14:46:44.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:46:44.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:44.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: CEPH_ARGS= 2021-08-21T14:46:44.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_recovery_reservations 2021-08-21T14:46:44.744 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:44.744 INFO:tasks.workunit.client.0.smithi077.stdout: "local_reservations": { 2021-08-21T14:46:44.745 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:44.745 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:44.745 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:44.745 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:44.745 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:44.746 INFO:tasks.workunit.client.0.smithi077.stdout: "remote_reservations": { 2021-08-21T14:46:44.746 INFO:tasks.workunit.client.0.smithi077.stdout: "max_allowed": 1, 2021-08-21T14:46:44.746 INFO:tasks.workunit.client.0.smithi077.stdout: "min_priority": 0, 2021-08-21T14:46:44.746 INFO:tasks.workunit.client.0.smithi077.stdout: "queues": [], 2021-08-21T14:46:44.747 INFO:tasks.workunit.client.0.smithi077.stdout: "in_progress": [] 2021-08-21T14:46:44.747 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:44.747 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:44.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:46:44.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:46:44.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:46:44.855 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:46:44.855 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:46:44.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:46:44.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:46:44.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:46:44.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:46:45.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=3 2021-08-21T14:46:45.217 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:46:45.218 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:46:45.218 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:46:45.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 3 = 3 2021-08-21T14:46:45.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:46:45.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:46:45.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:301: TEST_backfill_priority: ceph pg dump pgs 2021-08-21T14:46:46.097 INFO:tasks.workunit.client.0.smithi077.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2021-08-21T14:46:46.097 INFO:tasks.workunit.client.0.smithi077.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:45:47.459069+0000 93'150 118:341 [1,4] 1 [1,4] 1 0'0 2021-08-21T14:42:42.796484+0000 0'0 2021-08-21T14:42:42.796484+0000 0 2021-08-21T14:46:46.097 INFO:tasks.workunit.client.0.smithi077.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:46:40.383174+0000 93'150 119:350 [1,2] 1 [1,2] 1 0'0 2021-08-21T14:42:26.190001+0000 0'0 2021-08-21T14:42:26.190001+0000 0 2021-08-21T14:46:46.098 INFO:tasks.workunit.client.0.smithi077.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2021-08-21T14:46:13.843560+0000 93'150 118:357 [1,2] 1 [1,2] 1 0'0 2021-08-21T14:42:31.714089+0000 0'0 2021-08-21T14:42:31.714089+0000 0 2021-08-21T14:46:46.098 INFO:tasks.workunit.client.0.smithi077.stdout: 2021-08-21T14:46:46.098 INFO:tasks.workunit.client.0.smithi077.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2021-08-21T14:46:46.098 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:46:46.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:303: TEST_backfill_priority: get_asok_path osd.1 2021-08-21T14:46:46.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2021-08-21T14:46:46.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2021-08-21T14:46:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:46:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:46:46.110 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26783/ceph-osd.1.asok 2021-08-21T14:46:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:303: TEST_backfill_priority: CEPH_ARGS= 2021-08-21T14:46:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:303: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.26783/ceph-osd.1.asok dump_pgstate_history 2021-08-21T14:46:46.205 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:46:46.206 INFO:tasks.workunit.client.0.smithi077.stdout: "pgs": [ 2021-08-21T14:46:46.206 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.206 INFO:tasks.workunit.client.0.smithi077.stdout: "pg": "7.0", 2021-08-21T14:46:46.207 INFO:tasks.workunit.client.0.smithi077.stdout: "currently": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.207 INFO:tasks.workunit.client.0.smithi077.stdout: "history": [ 2021-08-21T14:46:46.207 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.207 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "63", 2021-08-21T14:46:46.208 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.208 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.208 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.208 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.199549+0000", 2021-08-21T14:46:46.209 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.199563+0000" 2021-08-21T14:46:46.209 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.209 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.209 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.210 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.199573+0000", 2021-08-21T14:46:46.210 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.299987+0000" 2021-08-21T14:46:46.210 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.210 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.211 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.211 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.299987+0000", 2021-08-21T14:46:46.211 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.300036+0000" 2021-08-21T14:46:46.211 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.211 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.212 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.212 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.300036+0000", 2021-08-21T14:46:46.212 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.300045+0000" 2021-08-21T14:46:46.212 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.213 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.213 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.213 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.300046+0000", 2021-08-21T14:46:46.213 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.408838+0000" 2021-08-21T14:46:46.214 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.214 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.214 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.214 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.199569+0000", 2021-08-21T14:46:46.214 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.408843+0000" 2021-08-21T14:46:46.215 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.215 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.215 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.215 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.408977+0000", 2021-08-21T14:46:46.216 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.946745+0000" 2021-08-21T14:46:46.216 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.216 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.216 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.217 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.946746+0000", 2021-08-21T14:46:46.217 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:43.946808+0000" 2021-08-21T14:46:46.217 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.217 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.217 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.218 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.946809+0000", 2021-08-21T14:46:46.218 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.649992+0000" 2021-08-21T14:46:46.218 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.218 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.219 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.219 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.408844+0000", 2021-08-21T14:46:46.219 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.650008+0000" 2021-08-21T14:46:46.220 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.220 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.220 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.220 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.199563+0000", 2021-08-21T14:46:46.221 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.650023+0000" 2021-08-21T14:46:46.221 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.221 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.221 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.221 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:43.199544+0000", 2021-08-21T14:46:46.222 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.650027+0000" 2021-08-21T14:46:46.222 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.222 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.222 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.223 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.223 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "63", 2021-08-21T14:46:46.223 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.223 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.224 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.224 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.650027+0000", 2021-08-21T14:46:46.224 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.650248+0000" 2021-08-21T14:46:46.224 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.225 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.225 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.225 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.225 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "91", 2021-08-21T14:46:46.225 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.226 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.226 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.226 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.650251+0000", 2021-08-21T14:46:46.226 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.650261+0000" 2021-08-21T14:46:46.227 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.227 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.227 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.227 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.650268+0000", 2021-08-21T14:46:46.228 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.743116+0000" 2021-08-21T14:46:46.228 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.228 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.228 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.228 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.743117+0000", 2021-08-21T14:46:46.229 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.743212+0000" 2021-08-21T14:46:46.229 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.230 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.231 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.231 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.743212+0000", 2021-08-21T14:46:46.232 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:44.743223+0000" 2021-08-21T14:46:46.232 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.232 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.233 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.233 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.743224+0000", 2021-08-21T14:46:46.234 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:45.019305+0000" 2021-08-21T14:46:46.234 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.235 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.235 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.235 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.650265+0000", 2021-08-21T14:46:46.235 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:45.019311+0000" 2021-08-21T14:46:46.235 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.236 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.236 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.236 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:45.019469+0000", 2021-08-21T14:46:46.236 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:45.991946+0000" 2021-08-21T14:46:46.237 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.237 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.237 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.237 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:45.991947+0000", 2021-08-21T14:46:46.238 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:45.991982+0000" 2021-08-21T14:46:46.238 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.238 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.239 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.239 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:45.991983+0000", 2021-08-21T14:46:46.239 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.308691+0000" 2021-08-21T14:46:46.239 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.240 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.240 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.240 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:45.019311+0000", 2021-08-21T14:46:46.240 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.308721+0000" 2021-08-21T14:46:46.241 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.241 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.241 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.241 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.650262+0000", 2021-08-21T14:46:46.242 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.308746+0000" 2021-08-21T14:46:46.242 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.242 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.242 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.243 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:44.650248+0000", 2021-08-21T14:46:46.243 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.308752+0000" 2021-08-21T14:46:46.243 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.243 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.243 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.244 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.244 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "92", 2021-08-21T14:46:46.244 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.244 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.244 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.245 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.308753+0000", 2021-08-21T14:46:46.245 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.309238+0000" 2021-08-21T14:46:46.245 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.245 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.246 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.246 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.246 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "98", 2021-08-21T14:46:46.246 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.246 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.247 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.247 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.309246+0000", 2021-08-21T14:46:46.247 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.309268+0000" 2021-08-21T14:46:46.247 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.247 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.248 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.248 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.309282+0000", 2021-08-21T14:46:46.248 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.410195+0000" 2021-08-21T14:46:46.248 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.249 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.249 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.249 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.410195+0000", 2021-08-21T14:46:46.250 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.410280+0000" 2021-08-21T14:46:46.250 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.250 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.250 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.250 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.410281+0000", 2021-08-21T14:46:46.251 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.410297+0000" 2021-08-21T14:46:46.251 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.251 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.251 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.251 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.410298+0000", 2021-08-21T14:46:46.252 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.887654+0000" 2021-08-21T14:46:46.252 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.252 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.252 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.253 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.309276+0000", 2021-08-21T14:46:46.253 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.887665+0000" 2021-08-21T14:46:46.253 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.254 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.254 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.254 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.887782+0000", 2021-08-21T14:46:46.255 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.938425+0000" 2021-08-21T14:46:46.255 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.255 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.255 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.255 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.938429+0000", 2021-08-21T14:46:46.256 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.938531+0000" 2021-08-21T14:46:46.256 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.256 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.256 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.256 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.938532+0000", 2021-08-21T14:46:46.257 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.053426+0000" 2021-08-21T14:46:46.257 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.257 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.257 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.257 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.887666+0000", 2021-08-21T14:46:46.258 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.053446+0000" 2021-08-21T14:46:46.258 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.258 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.258 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.259 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.309269+0000", 2021-08-21T14:46:46.259 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.053500+0000" 2021-08-21T14:46:46.259 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.259 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.259 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.260 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.309239+0000", 2021-08-21T14:46:46.260 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.053508+0000" 2021-08-21T14:46:46.260 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.260 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.260 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.261 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.261 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "98", 2021-08-21T14:46:46.261 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.262 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.262 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.263 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.053509+0000", 2021-08-21T14:46:46.263 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.053882+0000" 2021-08-21T14:46:46.263 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.264 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.264 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.264 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.265 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "101", 2021-08-21T14:46:46.265 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.265 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.265 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.266 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.053887+0000", 2021-08-21T14:46:46.266 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.053918+0000" 2021-08-21T14:46:46.266 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.266 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.266 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.267 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.053926+0000", 2021-08-21T14:46:46.267 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.170611+0000" 2021-08-21T14:46:46.267 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.267 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.268 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.268 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.170611+0000", 2021-08-21T14:46:46.268 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.170662+0000" 2021-08-21T14:46:46.268 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.268 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.269 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.269 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.053923+0000", 2021-08-21T14:46:46.269 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.170669+0000" 2021-08-21T14:46:46.269 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.269 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.270 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/WaitActingChange", 2021-08-21T14:46:46.270 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.170669+0000", 2021-08-21T14:46:46.270 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.522928+0000" 2021-08-21T14:46:46.270 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.271 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.271 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.271 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.053919+0000", 2021-08-21T14:46:46.271 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.522949+0000" 2021-08-21T14:46:46.271 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.272 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.272 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.272 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.053882+0000", 2021-08-21T14:46:46.272 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.522952+0000" 2021-08-21T14:46:46.272 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.273 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.273 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.273 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.273 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "101", 2021-08-21T14:46:46.274 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.274 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.274 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.274 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.522953+0000", 2021-08-21T14:46:46.274 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.523250+0000" 2021-08-21T14:46:46.275 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.275 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.275 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.275 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.276 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "114", 2021-08-21T14:46:46.276 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.276 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.276 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.277 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.523255+0000", 2021-08-21T14:46:46.277 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.523267+0000" 2021-08-21T14:46:46.277 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.277 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.278 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.278 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.523275+0000", 2021-08-21T14:46:46.278 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.615345+0000" 2021-08-21T14:46:46.279 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.279 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.279 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.279 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.615345+0000", 2021-08-21T14:46:46.280 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.615423+0000" 2021-08-21T14:46:46.280 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.280 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.281 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.281 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.615423+0000", 2021-08-21T14:46:46.281 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.615439+0000" 2021-08-21T14:46:46.281 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.281 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.282 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.282 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.615439+0000", 2021-08-21T14:46:46.282 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.816346+0000" 2021-08-21T14:46:46.282 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.283 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.283 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.283 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.523271+0000", 2021-08-21T14:46:46.283 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:32.816356+0000" 2021-08-21T14:46:46.283 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.284 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.284 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.284 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.816653+0000", 2021-08-21T14:46:46.284 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:33.360594+0000" 2021-08-21T14:46:46.284 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.285 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.285 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-08-21T14:46:46.285 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:33.360594+0000", 2021-08-21T14:46:46.285 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:33.360720+0000" 2021-08-21T14:46:46.286 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.286 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.286 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-08-21T14:46:46.286 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:33.360721+0000", 2021-08-21T14:46:46.286 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:33.361097+0000" 2021-08-21T14:46:46.287 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.287 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.287 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Backfilling", 2021-08-21T14:46:46.287 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:33.361098+0000", 2021-08-21T14:46:46.288 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:03.635965+0000" 2021-08-21T14:46:46.288 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.288 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.288 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/NotBackfilling", 2021-08-21T14:46:46.288 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:03.635966+0000", 2021-08-21T14:46:46.289 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:03.636144+0000" 2021-08-21T14:46:46.289 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.289 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.289 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-08-21T14:46:46.290 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:03.636145+0000", 2021-08-21T14:46:46.290 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:16.131283+0000" 2021-08-21T14:46:46.290 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.290 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.291 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-08-21T14:46:46.291 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:16.131283+0000", 2021-08-21T14:46:46.291 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:16.131630+0000" 2021-08-21T14:46:46.291 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.292 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.292 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Backfilling", 2021-08-21T14:46:46.292 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:16.131631+0000", 2021-08-21T14:46:46.292 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:45.951224+0000" 2021-08-21T14:46:46.292 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.293 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.293 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.293 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:45.951224+0000", 2021-08-21T14:46:46.293 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:45.951341+0000" 2021-08-21T14:46:46.294 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.294 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.294 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.294 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:45.951341+0000", 2021-08-21T14:46:46.295 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:46.419279+0000" 2021-08-21T14:46:46.295 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.295 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.295 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.295 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.816357+0000", 2021-08-21T14:46:46.296 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:46.419325+0000" 2021-08-21T14:46:46.296 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.296 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.296 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.297 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.523268+0000", 2021-08-21T14:46:46.297 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:46.419364+0000" 2021-08-21T14:46:46.297 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.297 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.297 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.298 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:32.523250+0000", 2021-08-21T14:46:46.298 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:46.419372+0000" 2021-08-21T14:46:46.298 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.298 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.299 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.299 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.299 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "114", 2021-08-21T14:46:46.299 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.299 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.300 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.300 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:46.419373+0000", 2021-08-21T14:46:46.300 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:46.419788+0000" 2021-08-21T14:46:46.300 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.301 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.301 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.301 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.301 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.301 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.302 INFO:tasks.workunit.client.0.smithi077.stdout: "pg": "3.0", 2021-08-21T14:46:46.302 INFO:tasks.workunit.client.0.smithi077.stdout: "currently": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.302 INFO:tasks.workunit.client.0.smithi077.stdout: "history": [ 2021-08-21T14:46:46.302 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.302 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "40", 2021-08-21T14:46:46.303 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.303 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.303 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.303 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.107628+0000", 2021-08-21T14:46:46.304 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:32.107636+0000" 2021-08-21T14:46:46.304 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.304 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.304 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.305 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.107642+0000", 2021-08-21T14:46:46.305 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:32.235216+0000" 2021-08-21T14:46:46.305 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.305 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.305 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.306 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.235216+0000", 2021-08-21T14:46:46.306 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:32.235264+0000" 2021-08-21T14:46:46.306 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.306 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.306 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.307 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.235264+0000", 2021-08-21T14:46:46.307 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:32.235273+0000" 2021-08-21T14:46:46.307 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.307 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.308 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.308 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.235274+0000", 2021-08-21T14:46:46.308 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:32.552287+0000" 2021-08-21T14:46:46.308 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.308 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.309 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.309 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.107640+0000", 2021-08-21T14:46:46.309 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:32.552293+0000" 2021-08-21T14:46:46.309 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.309 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.310 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.310 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.552431+0000", 2021-08-21T14:46:46.310 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.560208+0000" 2021-08-21T14:46:46.310 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.311 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.311 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.311 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.560209+0000", 2021-08-21T14:46:46.311 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.560275+0000" 2021-08-21T14:46:46.311 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.312 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.312 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.312 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.560276+0000", 2021-08-21T14:46:46.312 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.891618+0000" 2021-08-21T14:46:46.312 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.313 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.313 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.313 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.552293+0000", 2021-08-21T14:46:46.313 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.891634+0000" 2021-08-21T14:46:46.313 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.314 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.314 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.314 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.107637+0000", 2021-08-21T14:46:46.314 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.891652+0000" 2021-08-21T14:46:46.315 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.315 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.315 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.315 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:32.107626+0000", 2021-08-21T14:46:46.315 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.891655+0000" 2021-08-21T14:46:46.316 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.316 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.316 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.316 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.316 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "40", 2021-08-21T14:46:46.317 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.317 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.317 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.317 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.891656+0000", 2021-08-21T14:46:46.317 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.891902+0000" 2021-08-21T14:46:46.318 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.318 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.318 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.318 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.318 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "89", 2021-08-21T14:46:46.319 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.319 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.319 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.319 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.891906+0000", 2021-08-21T14:46:46.320 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:33.891918+0000" 2021-08-21T14:46:46.320 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.320 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.320 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.320 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.891925+0000", 2021-08-21T14:46:46.321 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.011771+0000" 2021-08-21T14:46:46.321 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.321 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.321 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.321 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.011771+0000", 2021-08-21T14:46:46.322 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.011857+0000" 2021-08-21T14:46:46.322 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.322 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.322 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.322 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.011858+0000", 2021-08-21T14:46:46.323 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.011874+0000" 2021-08-21T14:46:46.323 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.323 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.323 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.323 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.011874+0000", 2021-08-21T14:46:46.324 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.279759+0000" 2021-08-21T14:46:46.324 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.324 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.324 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.324 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.891922+0000", 2021-08-21T14:46:46.325 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.279767+0000" 2021-08-21T14:46:46.325 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.325 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.325 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.326 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.279903+0000", 2021-08-21T14:46:46.326 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.658147+0000" 2021-08-21T14:46:46.326 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.326 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.326 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.327 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.658148+0000", 2021-08-21T14:46:46.327 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:34.658187+0000" 2021-08-21T14:46:46.327 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.327 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.327 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.328 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.658188+0000", 2021-08-21T14:46:46.328 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.309559+0000" 2021-08-21T14:46:46.328 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.328 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.329 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.329 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:34.279768+0000", 2021-08-21T14:46:46.329 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.309583+0000" 2021-08-21T14:46:46.329 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.329 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.330 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.330 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.891919+0000", 2021-08-21T14:46:46.330 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.309606+0000" 2021-08-21T14:46:46.330 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.331 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.331 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.331 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:33.891902+0000", 2021-08-21T14:46:46.331 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.309610+0000" 2021-08-21T14:46:46.331 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.332 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.332 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.332 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.332 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "92", 2021-08-21T14:46:46.332 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.333 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.333 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.333 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.309611+0000", 2021-08-21T14:46:46.333 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.310096+0000" 2021-08-21T14:46:46.333 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.334 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.334 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.334 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.334 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "106", 2021-08-21T14:46:46.335 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.335 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.335 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.335 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.310103+0000", 2021-08-21T14:46:46.335 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.310122+0000" 2021-08-21T14:46:46.336 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.336 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.336 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.336 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.310136+0000", 2021-08-21T14:46:46.336 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.410411+0000" 2021-08-21T14:46:46.337 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.337 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.337 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.337 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.410412+0000", 2021-08-21T14:46:46.338 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.410453+0000" 2021-08-21T14:46:46.338 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.338 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.338 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.338 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.410453+0000", 2021-08-21T14:46:46.339 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.410461+0000" 2021-08-21T14:46:46.339 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.339 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.339 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.339 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.410462+0000", 2021-08-21T14:46:46.340 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.887960+0000" 2021-08-21T14:46:46.340 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.340 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.340 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.340 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.310129+0000", 2021-08-21T14:46:46.341 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:13.887970+0000" 2021-08-21T14:46:46.341 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.341 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.341 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.342 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.888080+0000", 2021-08-21T14:46:46.342 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:14.001434+0000" 2021-08-21T14:46:46.342 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.342 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.342 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.343 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:14.001434+0000", 2021-08-21T14:46:46.343 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:43:14.001468+0000" 2021-08-21T14:46:46.343 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.343 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.343 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.344 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:14.001468+0000", 2021-08-21T14:46:46.344 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.275605+0000" 2021-08-21T14:46:46.344 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.344 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.344 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.345 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.887970+0000", 2021-08-21T14:46:46.345 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.275639+0000" 2021-08-21T14:46:46.345 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.345 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.346 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.346 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.310123+0000", 2021-08-21T14:46:46.346 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.275702+0000" 2021-08-21T14:46:46.346 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.346 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.347 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.347 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:43:13.310096+0000", 2021-08-21T14:46:46.347 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.275706+0000" 2021-08-21T14:46:46.347 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.348 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.348 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.348 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.348 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "107", 2021-08-21T14:46:46.348 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.349 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.349 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.349 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.275707+0000", 2021-08-21T14:46:46.349 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.276167+0000" 2021-08-21T14:46:46.349 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.350 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.350 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.350 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.350 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "108", 2021-08-21T14:46:46.350 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.351 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.351 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.351 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.276172+0000", 2021-08-21T14:46:46.351 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.276200+0000" 2021-08-21T14:46:46.352 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.352 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.352 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.352 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.276211+0000", 2021-08-21T14:46:46.352 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.297011+0000" 2021-08-21T14:46:46.353 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.353 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.353 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.353 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.297012+0000", 2021-08-21T14:46:46.353 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.297067+0000" 2021-08-21T14:46:46.354 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.354 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.354 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.354 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.276207+0000", 2021-08-21T14:46:46.354 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.297073+0000" 2021-08-21T14:46:46.355 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.355 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.355 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/WaitActingChange", 2021-08-21T14:46:46.355 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.297074+0000", 2021-08-21T14:46:46.356 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.594161+0000" 2021-08-21T14:46:46.356 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.356 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.356 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.356 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.276201+0000", 2021-08-21T14:46:46.357 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.594189+0000" 2021-08-21T14:46:46.357 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.357 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.357 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.357 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.276168+0000", 2021-08-21T14:46:46.358 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.594193+0000" 2021-08-21T14:46:46.358 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.358 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.358 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.358 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.359 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "108", 2021-08-21T14:46:46.359 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.359 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.359 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.360 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.594194+0000", 2021-08-21T14:46:46.360 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.594456+0000" 2021-08-21T14:46:46.360 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.360 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.360 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.361 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.361 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "116", 2021-08-21T14:46:46.361 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.361 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.361 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.362 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.594460+0000", 2021-08-21T14:46:46.362 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.594471+0000" 2021-08-21T14:46:46.362 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.362 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.363 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.363 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.594479+0000", 2021-08-21T14:46:46.363 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.694908+0000" 2021-08-21T14:46:46.363 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.363 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.364 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.364 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.694908+0000", 2021-08-21T14:46:46.364 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.694980+0000" 2021-08-21T14:46:46.364 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.364 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.365 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.365 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.694980+0000", 2021-08-21T14:46:46.365 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.694996+0000" 2021-08-21T14:46:46.365 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.366 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.366 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.366 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.694997+0000", 2021-08-21T14:46:46.366 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.963221+0000" 2021-08-21T14:46:46.367 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.367 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.367 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.367 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.594476+0000", 2021-08-21T14:46:46.367 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:50.963232+0000" 2021-08-21T14:46:46.368 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.368 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.368 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.368 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.963495+0000", 2021-08-21T14:46:46.368 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:51.114425+0000" 2021-08-21T14:46:46.369 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.369 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.369 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-08-21T14:46:46.369 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:51.114425+0000", 2021-08-21T14:46:46.370 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:03.636067+0000" 2021-08-21T14:46:46.370 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.370 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.370 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-08-21T14:46:46.370 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:03.636067+0000", 2021-08-21T14:46:46.371 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:03.636386+0000" 2021-08-21T14:46:46.371 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.371 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.371 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Backfilling", 2021-08-21T14:46:46.371 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:03.636386+0000", 2021-08-21T14:46:46.372 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:16.131227+0000" 2021-08-21T14:46:46.372 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.372 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.372 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/NotBackfilling", 2021-08-21T14:46:46.373 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:16.131227+0000", 2021-08-21T14:46:46.373 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:16.131388+0000" 2021-08-21T14:46:46.373 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.373 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.373 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-08-21T14:46:46.374 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:16.131388+0000", 2021-08-21T14:46:46.374 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:45.951480+0000" 2021-08-21T14:46:46.374 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.374 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.375 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-08-21T14:46:46.375 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:45.951481+0000", 2021-08-21T14:46:46.375 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:45:45.951858+0000" 2021-08-21T14:46:46.375 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.375 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.376 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Backfilling", 2021-08-21T14:46:46.376 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:45:45.951858+0000", 2021-08-21T14:46:46.376 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.335391+0000" 2021-08-21T14:46:46.376 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.377 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.377 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.377 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:12.335391+0000", 2021-08-21T14:46:46.377 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.335487+0000" 2021-08-21T14:46:46.377 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.378 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.378 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.378 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:12.335487+0000", 2021-08-21T14:46:46.378 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.896560+0000" 2021-08-21T14:46:46.378 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.379 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.379 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.379 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.963232+0000", 2021-08-21T14:46:46.379 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.896592+0000" 2021-08-21T14:46:46.379 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.380 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.380 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.380 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.594472+0000", 2021-08-21T14:46:46.380 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.896614+0000" 2021-08-21T14:46:46.381 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.381 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.381 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.381 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:50.594456+0000", 2021-08-21T14:46:46.381 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.896618+0000" 2021-08-21T14:46:46.382 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.382 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.382 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.382 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.383 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "116", 2021-08-21T14:46:46.383 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.383 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.383 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.383 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:12.896619+0000", 2021-08-21T14:46:46.384 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.897002+0000" 2021-08-21T14:46:46.384 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.384 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.384 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.384 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.385 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.385 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.385 INFO:tasks.workunit.client.0.smithi077.stdout: "pg": "1.0", 2021-08-21T14:46:46.385 INFO:tasks.workunit.client.0.smithi077.stdout: "currently": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.385 INFO:tasks.workunit.client.0.smithi077.stdout: "history": [ 2021-08-21T14:46:46.386 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.386 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "27", 2021-08-21T14:46:46.386 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.386 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.386 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Initial", 2021-08-21T14:46:46.387 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127716+0000", 2021-08-21T14:46:46.387 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.127868+0000" 2021-08-21T14:46:46.387 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.387 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.388 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.388 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.388 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "27", 2021-08-21T14:46:46.388 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.388 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.389 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.389 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127869+0000", 2021-08-21T14:46:46.389 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.127901+0000" 2021-08-21T14:46:46.389 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.389 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.390 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.390 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.390 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "29", 2021-08-21T14:46:46.390 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.391 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.391 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.391 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127906+0000", 2021-08-21T14:46:46.391 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.127919+0000" 2021-08-21T14:46:46.391 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.392 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.392 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.392 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127926+0000", 2021-08-21T14:46:46.392 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.128468+0000" 2021-08-21T14:46:46.392 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.393 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.393 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.393 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.128468+0000", 2021-08-21T14:46:46.393 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.128559+0000" 2021-08-21T14:46:46.394 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.394 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.394 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.394 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.128560+0000", 2021-08-21T14:46:46.394 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.128570+0000" 2021-08-21T14:46:46.395 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.395 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.395 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.395 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.128570+0000", 2021-08-21T14:46:46.396 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.341065+0000" 2021-08-21T14:46:46.396 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.396 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.396 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.396 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127923+0000", 2021-08-21T14:46:46.397 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:27.341074+0000" 2021-08-21T14:46:46.397 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.397 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.397 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.398 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.341255+0000", 2021-08-21T14:46:46.398 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.073766+0000" 2021-08-21T14:46:46.398 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.398 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.398 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.399 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.341082+0000", 2021-08-21T14:46:46.399 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.073823+0000" 2021-08-21T14:46:46.399 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.399 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.399 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.400 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127920+0000", 2021-08-21T14:46:46.400 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.073887+0000" 2021-08-21T14:46:46.400 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.400 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.401 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.401 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:27.127901+0000", 2021-08-21T14:46:46.401 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.073911+0000" 2021-08-21T14:46:46.401 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.401 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.402 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.402 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.402 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "29", 2021-08-21T14:46:46.402 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.402 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.403 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.403 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.073912+0000", 2021-08-21T14:46:46.403 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.074141+0000" 2021-08-21T14:46:46.403 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.403 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.404 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.404 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.404 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "103", 2021-08-21T14:46:46.404 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.405 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.405 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.405 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.074144+0000", 2021-08-21T14:46:46.405 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.074151+0000" 2021-08-21T14:46:46.405 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.406 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.406 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.406 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.074157+0000", 2021-08-21T14:46:46.406 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.275136+0000" 2021-08-21T14:46:46.407 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.407 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.407 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.407 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.275137+0000", 2021-08-21T14:46:46.407 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.275227+0000" 2021-08-21T14:46:46.408 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.408 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.408 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.408 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.275228+0000", 2021-08-21T14:46:46.408 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.275243+0000" 2021-08-21T14:46:46.409 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.409 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.409 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.409 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.275243+0000", 2021-08-21T14:46:46.409 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.694390+0000" 2021-08-21T14:46:46.410 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.410 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.410 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.410 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.074154+0000", 2021-08-21T14:46:46.411 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.694397+0000" 2021-08-21T14:46:46.411 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.411 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.411 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.411 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.694534+0000", 2021-08-21T14:46:46.412 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.886818+0000" 2021-08-21T14:46:46.412 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.412 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.412 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.413 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.886818+0000", 2021-08-21T14:46:46.413 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:42:28.886855+0000" 2021-08-21T14:46:46.413 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.413 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.413 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.414 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.886856+0000", 2021-08-21T14:46:46.414 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.423126+0000" 2021-08-21T14:46:46.414 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.414 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.414 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.415 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.694397+0000", 2021-08-21T14:46:46.415 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.423157+0000" 2021-08-21T14:46:46.415 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.415 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.415 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.416 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.074152+0000", 2021-08-21T14:46:46.416 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.423224+0000" 2021-08-21T14:46:46.416 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.416 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.417 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.417 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:42:28.074142+0000", 2021-08-21T14:46:46.417 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.423229+0000" 2021-08-21T14:46:46.417 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.417 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.418 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.418 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.418 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "103", 2021-08-21T14:46:46.418 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.418 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.419 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.419 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.423229+0000", 2021-08-21T14:46:46.419 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.423679+0000" 2021-08-21T14:46:46.419 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.420 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.420 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.420 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.420 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "104", 2021-08-21T14:46:46.420 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.421 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.421 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.421 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.423685+0000", 2021-08-21T14:46:46.421 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.423700+0000" 2021-08-21T14:46:46.422 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.422 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.422 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.422 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.423711+0000", 2021-08-21T14:46:46.422 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.424290+0000" 2021-08-21T14:46:46.423 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.423 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.423 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.423 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.424290+0000", 2021-08-21T14:46:46.423 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.424367+0000" 2021-08-21T14:46:46.424 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.424 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.424 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.424 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.423705+0000", 2021-08-21T14:46:46.425 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.424374+0000" 2021-08-21T14:46:46.425 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.425 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.425 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/WaitActingChange", 2021-08-21T14:46:46.425 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.424375+0000", 2021-08-21T14:46:46.426 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.641458+0000" 2021-08-21T14:46:46.426 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.426 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.426 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.426 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.423701+0000", 2021-08-21T14:46:46.427 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.641476+0000" 2021-08-21T14:46:46.427 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.427 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.427 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.427 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.423680+0000", 2021-08-21T14:46:46.428 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.641479+0000" 2021-08-21T14:46:46.428 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.428 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.428 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.429 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.429 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "104", 2021-08-21T14:46:46.429 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.429 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.429 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.430 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.641480+0000", 2021-08-21T14:46:46.430 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.641730+0000" 2021-08-21T14:46:46.430 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.430 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.430 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.431 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.431 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "118", 2021-08-21T14:46:46.431 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.431 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.432 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Start", 2021-08-21T14:46:46.432 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.641735+0000", 2021-08-21T14:46:46.432 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.641765+0000" 2021-08-21T14:46:46.432 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.432 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.433 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetInfo", 2021-08-21T14:46:46.433 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.641771+0000", 2021-08-21T14:46:46.433 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.759373+0000" 2021-08-21T14:46:46.433 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.434 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.434 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetLog", 2021-08-21T14:46:46.434 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.759373+0000", 2021-08-21T14:46:46.434 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.759452+0000" 2021-08-21T14:46:46.434 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.435 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.435 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/GetMissing", 2021-08-21T14:46:46.435 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.759452+0000", 2021-08-21T14:46:46.435 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:44.759468+0000" 2021-08-21T14:46:46.435 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.436 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.436 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2021-08-21T14:46:46.436 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.759469+0000", 2021-08-21T14:46:46.436 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:45.060722+0000" 2021-08-21T14:46:46.437 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.437 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.437 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Peering", 2021-08-21T14:46:46.437 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.641768+0000", 2021-08-21T14:46:46.437 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:45.060732+0000" 2021-08-21T14:46:46.438 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.438 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.438 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Activating", 2021-08-21T14:46:46.438 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:45.060975+0000", 2021-08-21T14:46:46.439 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:44:45.396818+0000" 2021-08-21T14:46:46.439 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.439 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.439 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2021-08-21T14:46:46.439 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:45.396819+0000", 2021-08-21T14:46:46.440 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.335615+0000" 2021-08-21T14:46:46.440 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.440 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.440 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2021-08-21T14:46:46.440 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:12.335615+0000", 2021-08-21T14:46:46.441 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:12.335898+0000" 2021-08-21T14:46:46.441 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.441 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.441 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Backfilling", 2021-08-21T14:46:46.442 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:12.335898+0000", 2021-08-21T14:46:46.442 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.519184+0000" 2021-08-21T14:46:46.442 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.442 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.442 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Recovered", 2021-08-21T14:46:46.443 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:39.519184+0000", 2021-08-21T14:46:46.443 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.519411+0000" 2021-08-21T14:46:46.443 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.443 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.444 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active/Clean", 2021-08-21T14:46:46.444 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:39.519412+0000", 2021-08-21T14:46:46.444 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.787928+0000" 2021-08-21T14:46:46.444 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.445 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.445 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary/Active", 2021-08-21T14:46:46.445 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:45.060732+0000", 2021-08-21T14:46:46.445 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.787946+0000" 2021-08-21T14:46:46.445 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.446 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.446 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started/Primary", 2021-08-21T14:46:46.446 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.641765+0000", 2021-08-21T14:46:46.446 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.787964+0000" 2021-08-21T14:46:46.447 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.447 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.447 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Started", 2021-08-21T14:46:46.447 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:44:44.641730+0000", 2021-08-21T14:46:46.447 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.787968+0000" 2021-08-21T14:46:46.448 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.448 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.448 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:46:46.448 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.448 INFO:tasks.workunit.client.0.smithi077.stdout: "epoch": "118", 2021-08-21T14:46:46.449 INFO:tasks.workunit.client.0.smithi077.stdout: "states": [ 2021-08-21T14:46:46.449 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:46:46.449 INFO:tasks.workunit.client.0.smithi077.stdout: "state": "Reset", 2021-08-21T14:46:46.449 INFO:tasks.workunit.client.0.smithi077.stdout: "enter": "2021-08-21T14:46:39.787968+0000", 2021-08-21T14:46:46.449 INFO:tasks.workunit.client.0.smithi077.stdout: "exit": "2021-08-21T14:46:39.788289+0000" 2021-08-21T14:46:46.450 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.450 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.450 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.450 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.451 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:46:46.451 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:46:46.451 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:46:46.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:305: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2021-08-21T14:46:46.452 INFO:tasks.workunit.client.0.smithi077.stdout:TEST PASSED 2021-08-21T14:46:46.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:309: TEST_backfill_priority: echo TEST PASSED 2021-08-21T14:46:46.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:312: TEST_backfill_priority: delete_pool test1 2021-08-21T14:46:46.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2021-08-21T14:46:46.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2021-08-21T14:46:46.732 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test1' does not exist 2021-08-21T14:46:46.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:313: TEST_backfill_priority: delete_pool test3 2021-08-21T14:46:46.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2021-08-21T14:46:46.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2021-08-21T14:46:47.361 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test3' does not exist 2021-08-21T14:46:47.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:314: TEST_backfill_priority: delete_pool test7 2021-08-21T14:46:47.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2021-08-21T14:46:47.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2021-08-21T14:46:48.101 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test7' does not exist 2021-08-21T14:46:48.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:315: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2021-08-21T14:46:48.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:46:48.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:46:48.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:46:48.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:46:48.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:46:48.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:46:48.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:316: TEST_backfill_priority: return 0 2021-08-21T14:46:48.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:43: run: teardown td/osd-backfill-prio 2021-08-21T14:46:48.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-08-21T14:46:48.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:46:48.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-08-21T14:46:48.421 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:46:48.421 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:46:48.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:46:48.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:46:48.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:46:48.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:46:48.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:46:48.426 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:46:48.426 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:46:48.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:46:48.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:46:48.427 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:46:48.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:46:48.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:46:48.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:46:48.430 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:46:48.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:46:48.431 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:46:48.431 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-08-21T14:46:48.459 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:46:48.459 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:48.459 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:46:48.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26783 2021-08-21T14:46:48.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:46:48.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:46:48.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: main: code=0 2021-08-21T14:46:48.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: main: teardown td/osd-backfill-prio 0 2021-08-21T14:46:48.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2021-08-21T14:46:48.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2021-08-21T14:46:48.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2021-08-21T14:46:48.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:46:48.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:46:48.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:46:48.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:46:48.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:46:48.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:46:48.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:46:48.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:46:48.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:46:48.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:46:48.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:46:48.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:46:48.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:46:48.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:46:48.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:46:48.468 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:46:48.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:46:48.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2021-08-21T14:46:48.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2021-08-21T14:46:48.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:46:48.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:48.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26783 2021-08-21T14:46:48.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26783 2021-08-21T14:46:48.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:46:48.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:46:48.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: main: return 0 2021-08-21T14:46:48.473 INFO:teuthology.orchestra.run:Running command with timeout 3600 2021-08-21T14:46:48.473 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2021-08-21T14:46:48.507 INFO:tasks.workunit:Running workunit osd-backfill/osd-backfill-recovery-log.sh... 2021-08-21T14:46:48.507 DEBUG:teuthology.orchestra.run.smithi077:workunit test osd-backfill/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=cae1ffd55a16af4627c36508fde05bf24312fad6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh 2021-08-21T14:46:48.567 INFO:tasks.workunit.client.0.smithi077.stderr:stty: 'standard input': Inappropriate ioctl for device 2021-08-21T14:46:48.577 INFO:tasks.workunit.client.0.smithi077.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2021-08-21T14:46:48.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-08-21T14:46:48.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2021-08-21T14:46:48.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: main: export PYTHONWARNINGS=ignore 2021-08-21T14:46:48.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: main: PYTHONWARNINGS=ignore 2021-08-21T14:46:48.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: main: export CEPH_CONF=/dev/null 2021-08-21T14:46:48.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: main: CEPH_CONF=/dev/null 2021-08-21T14:46:48.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: main: unset CEPH_ARGS 2021-08-21T14:46:48.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: main: local code 2021-08-21T14:46:48.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: main: run td/osd-backfill-recovery-log 2021-08-21T14:46:48.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:22: run: shift 2021-08-21T14:46:48.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2021-08-21T14:46:48.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2021-08-21T14:46:48.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2021-08-21T14:46:48.580 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:27: run: uuidgen 2021-08-21T14:46:48.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none ' 2021-08-21T14:46:48.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:46:48.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:30: run: set 2021-08-21T14:46:48.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2021-08-21T14:46:48.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2021-08-21T14:46:48.582 INFO:tasks.workunit.client.0.smithi077.stderr:TEST_backfill_log_2 2021-08-21T14:46:48.582 INFO:tasks.workunit.client.0.smithi077.stderr:TEST_recovery_1 2021-08-21T14:46:48.582 INFO:tasks.workunit.client.0.smithi077.stderr:TEST_recovery_2' 2021-08-21T14:46:48.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-08-21T14:46:48.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-08-21T14:46:48.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-08-21T14:46:48.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:46:48.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-08-21T14:46:48.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:46:48.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:46:48.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:46:48.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:46:48.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:46:48.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:46:48.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:46:48.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:46:48.592 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:46:48.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:46:48.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:46:48.592 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:46:48.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:46:48.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:46:48.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:46:48.593 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:46:48.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:46:48.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:46:48.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-08-21T14:46:48.594 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:46:48.595 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:48.595 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:48.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53763 2021-08-21T14:46:48.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:46:48.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:46:48.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-08-21T14:46:48.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-08-21T14:46:48.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:48.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:48.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53763 2021-08-21T14:46:48.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-08-21T14:46:48.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2021-08-21T14:46:48.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2021-08-21T14:46:48.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-08-21T14:46:48.598 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-08-21T14:46:48.598 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2021-08-21T14:46:48.598 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.598 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:107: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2021-08-21T14:46:48.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:46:48.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2021-08-21T14:46:48.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2021-08-21T14:46:48.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2021-08-21T14:46:48.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2021-08-21T14:46:48.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-08-21T14:46:48.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-08-21T14:46:48.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-08-21T14:46:48.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-08-21T14:46:48.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-08-21T14:46:48.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-08-21T14:46:48.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-08-21T14:46:48.949 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-08-21T14:46:48.950 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:46:48.950 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:46:48.950 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:46:48.950 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:48.951 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:48.951 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:46:48.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-08-21T14:46:49.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-08-21T14:46:49.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-08-21T14:46:49.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:46:49.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:46:49.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=fsid 2021-08-21T14:46:49.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .fsid 2021-08-21T14:46:49.212 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:46:49.212 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:46:49.213 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:46:49.213 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:46:49.213 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:49.213 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:49.213 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53763/ceph-mon.a.asok 2021-08-21T14:46:49.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:46:49.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.53763/ceph-mon.a.asok config get fsid 2021-08-21T14:46:49.308 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-08-21T14:46:49.308 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:46:49.308 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:46:49.308 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=mon_host 2021-08-21T14:46:49.309 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .mon_host 2021-08-21T14:46:49.309 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:46:49.309 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:46:49.309 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:46:49.310 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:46:49.310 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:49.310 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:49.310 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53763/ceph-mon.a.asok 2021-08-21T14:46:49.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:46:49.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.53763/ceph-mon.a.asok config get mon_host 2021-08-21T14:46:49.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-08-21T14:46:49.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:49.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-08-21T14:46:49.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-08-21T14:46:49.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-08-21T14:46:49.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-08-21T14:46:49.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-08-21T14:46:50.290 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-08-21T14:46:50.290 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:46:50.291 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:46:50.291 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:46:50.291 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:50.291 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:50.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:46:50.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:46:50.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:46:50.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-08-21T14:46:50.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:46:50.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:46:50.316 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-08-21T14:46:50.316 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-08-21T14:46:50.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:46:50.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-08-21T14:46:50.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:46:50.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:46:50.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-08-21T14:46:50.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:46:50.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-08-21T14:46:50.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:46:50.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:46:50.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:46:50.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:46:50.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-08-21T14:46:50.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-08-21T14:46:50.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:46:50.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:46:50.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:46:50.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:46:50.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:46:50.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:46:50.325 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:46:50.325 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:46:50.326 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:46:50.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:46:50.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:46:50.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:46:50.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:46:50.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:46:50.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:46:50.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:46:50.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:46:50.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:46:50.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:46:50.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:46:50.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:46:50.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-08-21T14:46:50.329 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:46:50.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=2087a00e-9b9d-4832-ab68-2bed3b84d32c 2021-08-21T14:46:50.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 2087a00e-9b9d-4832-ab68-2bed3b84d32c' 2021-08-21T14:46:50.330 INFO:tasks.workunit.client.0.smithi077.stdout:add osd0 2087a00e-9b9d-4832-ab68-2bed3b84d32c 2021-08-21T14:46:50.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:46:50.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDaESFhwtzvExAAAQ9BF9NS4eglGCUKqeaw4g== 2021-08-21T14:46:50.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDaESFhwtzvExAAAQ9BF9NS4eglGCUKqeaw4g=="}' 2021-08-21T14:46:50.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 2087a00e-9b9d-4832-ab68-2bed3b84d32c -i td/osd-backfill-recovery-log/0/new.json 2021-08-21T14:46:50.685 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:46:50.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-08-21T14:46:50.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDaESFhwtzvExAAAQ9BF9NS4eglGCUKqeaw4g== --osd-uuid 2087a00e-9b9d-4832-ab68-2bed3b84d32c 2021-08-21T14:46:50.723 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.722+0000 7fbc12edc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:46:50.724 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.724+0000 7fbc12edc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:46:50.725 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.724+0000 7fbc12edc0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:46:50.726 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.725+0000 7fbc12edc0c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T14:46:50.726 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.725+0000 7fbc12edc0c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T14:46:50.727 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.725+0000 7fbc12edc0c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T14:46:50.826 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:50.825+0000 7fbc12edc0c0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-08-21T14:46:56.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-08-21T14:46:56.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:46:56.484 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd0 key to auth repository 2021-08-21T14:46:56.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-08-21T14:46:56.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:46:57.017 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.0 2021-08-21T14:46:57.018 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-08-21T14:46:57.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:46:57.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:46:57.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:46:57.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:46:57.042 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:57.041+0000 7fe938dae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:46:57.064 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:57.063+0000 7fe938dae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:46:57.065 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:57.064+0000 7fe938dae0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:46:57.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-08-21T14:46:57.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:46:57.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=0 2021-08-21T14:46:57.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:46:57.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:46:57.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:46:57.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:46:57.447 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:46:57.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:46:57.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:46:57.589 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:46:57.588+0000 7fe938dae0c0 -1 Falling back to public interface 2021-08-21T14:46:57.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:46:58.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:46:58.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:46:58.821 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:46:58.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:46:58.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:46:58.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:46:59.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:00.017 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:00.016+0000 7fe938dae0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-08-21T14:47:00.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:00.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:00.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:47:00.274 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:47:00.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:00.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:00.682 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:01.683 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:01.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:01.684 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:47:01.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:47:01.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:01.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:02.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:03.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:03.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:03.088 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:47:03.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:47:03.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:03.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:03.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:04.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:04.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:04.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:47:04.469 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:47:04.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:04.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:04.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:05.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:05.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:05.893 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:47:05.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:47:05.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:05.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:06.288 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:07.289 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:07.289 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:07.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:47:07.290 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:47:07.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:07.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:07.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:08.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:08.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:08.715 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:47:08.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:47:08.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:08.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:09.096 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:10.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:10.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:10.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:47:10.098 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:47:10.099 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:10.099 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:10.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:11.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:11.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:11.481 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:47:11.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:47:11.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:11.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:12.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:13.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:13.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:13.320 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:47:13.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:47:13.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:13.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:13.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:14.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:14.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:14.776 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:47:14.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:47:14.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:14.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:15.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:16.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:16.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:16.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:47:16.275 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:47:16.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:16.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:16.707 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:17.709 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:17.709 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:17.710 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:47:17.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:47:17.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:17.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:18.256 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:19.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:19.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:19.258 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:47:19.258 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:47:19.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:19.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:19.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:20.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:20.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:20.930 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:47:20.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:47:20.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:20.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:21.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:22.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:22.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:22.327 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:47:22.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:47:22.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:22.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:47:22.745 INFO:tasks.workunit.client.0.smithi077.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/54887,v1:127.0.0.1:6803/54887] [v2:127.0.0.1:6804/54887,v1:127.0.0.1:6805/54887] exists,up 2087a00e-9b9d-4832-ab68-2bed3b84d32c 2021-08-21T14:47:22.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:47:22.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:47:22.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:47:22.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:47:22.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-08-21T14:47:22.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:47:22.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:47:22.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-08-21T14:47:22.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:47:22.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-08-21T14:47:22.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:47:22.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:47:22.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:47:22.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:47:22.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-08-21T14:47:22.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-08-21T14:47:22.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:47:22.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:47:22.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:47:22.755 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:47:22.755 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:47:22.755 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:47:22.755 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:47:22.755 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:47:22.756 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:47:22.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:47:22.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:47:22.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:47:22.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:47:22.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:47:22.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:47:22.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:47:22.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:47:22.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:47:22.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:47:22.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:47:22.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:47:22.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-08-21T14:47:22.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:47:22.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=8ea001cc-d036-4e52-a71f-88c0326aff61 2021-08-21T14:47:22.764 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 8ea001cc-d036-4e52-a71f-88c0326aff61' 2021-08-21T14:47:22.764 INFO:tasks.workunit.client.0.smithi077.stdout:add osd1 8ea001cc-d036-4e52-a71f-88c0326aff61 2021-08-21T14:47:22.765 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:47:22.765 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQD6ESFhsRN2LRAAqoSaSepCdBHi8LOtYnNVjA== 2021-08-21T14:47:22.765 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQD6ESFhsRN2LRAAqoSaSepCdBHi8LOtYnNVjA=="}' 2021-08-21T14:47:22.766 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 8ea001cc-d036-4e52-a71f-88c0326aff61 -i td/osd-backfill-recovery-log/1/new.json 2021-08-21T14:47:23.414 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:47:23.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-08-21T14:47:23.426 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD6ESFhsRN2LRAAqoSaSepCdBHi8LOtYnNVjA== --osd-uuid 8ea001cc-d036-4e52-a71f-88c0326aff61 2021-08-21T14:47:23.452 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.450+0000 7fbfdb5b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:23.453 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.452+0000 7fbfdb5b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:23.454 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.453+0000 7fbfdb5b50c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:23.455 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.453+0000 7fbfdb5b50c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T14:47:23.455 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.453+0000 7fbfdb5b50c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T14:47:23.455 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.453+0000 7fbfdb5b50c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T14:47:23.526 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:23.525+0000 7fbfdb5b50c0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-08-21T14:47:29.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-08-21T14:47:29.053 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:47:29.054 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd1 key to auth repository 2021-08-21T14:47:29.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-08-21T14:47:29.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:47:29.592 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.1 2021-08-21T14:47:29.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-08-21T14:47:29.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:47:29.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:47:29.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:47:29.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:47:29.616 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:29.615+0000 7f0a7c65f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:29.649 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:29.648+0000 7f0a7c65f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:29.650 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:29.649+0000 7f0a7c65f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:29.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-08-21T14:47:29.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:47:29.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=1 2021-08-21T14:47:29.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:47:29.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:47:29.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:29.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:47:29.973 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:47:29.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:29.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:30.174 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:30.172+0000 7f0a7c65f0c0 -1 Falling back to public interface 2021-08-21T14:47:30.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:31.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:31.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:31.382 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:47:31.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:47:31.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:31.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:31.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:32.595 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:32.593+0000 7f0a7c65f0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-08-21T14:47:32.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:32.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:32.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:47:32.763 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:47:32.764 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:32.764 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:33.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:34.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:34.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:34.373 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:47:34.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:47:34.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:34.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:34.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:35.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:35.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:35.756 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:47:35.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:47:35.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:35.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:36.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:37.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:37.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:37.158 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:47:37.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:47:37.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:37.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:37.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:38.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:38.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:38.539 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:47:38.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:47:38.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:38.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:38.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:39.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:39.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:39.935 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:47:39.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:47:39.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:39.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:40.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:41.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:41.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:41.401 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:47:41.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:47:41.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:41.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:41.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:42.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:42.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:42.926 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:47:42.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:47:42.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:42.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:43.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:44.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:44.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:44.379 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:47:44.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:47:44.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:44.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:45.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:46.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:46.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:46.024 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:47:46.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:47:46.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:46.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:47.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:48.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:48.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:48.086 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:47:48.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:47:48.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:48.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:48.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:49.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:49.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:49.880 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:47:49.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:47:49.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:49.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:50.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:51.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:51.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:51.994 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:47:51.994 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:47:51.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:51.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:52.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:53.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:53.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:53.419 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:47:53.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:47:53.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:53.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:54.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:47:55.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:47:55.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:47:55.206 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:47:55.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:47:55.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:47:55.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:47:55.626 INFO:tasks.workunit.client.0.smithi077.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56670,v1:127.0.0.1:6811/56670] [v2:127.0.0.1:6812/56670,v1:127.0.0.1:6813/56670] exists,up 8ea001cc-d036-4e52-a71f-88c0326aff61 2021-08-21T14:47:55.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:47:55.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:47:55.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:47:55.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:47:55.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-08-21T14:47:55.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:47:55.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:47:55.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-08-21T14:47:55.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:47:55.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-08-21T14:47:55.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:47:55.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:47:55.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:47:55.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:47:55.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-08-21T14:47:55.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-08-21T14:47:55.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:47:55.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:47:55.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:47:55.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:47:55.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:47:55.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:47:55.635 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:47:55.635 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:47:55.636 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:47:55.636 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:47:55.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:47:55.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:47:55.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:47:55.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:47:55.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:47:55.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:47:55.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:47:55.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:47:55.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:47:55.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:47:55.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:47:55.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-08-21T14:47:55.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:47:55.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bfa2bbb2-d3db-4b40-b51f-6142a5ebf004 2021-08-21T14:47:55.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 bfa2bbb2-d3db-4b40-b51f-6142a5ebf004' 2021-08-21T14:47:55.639 INFO:tasks.workunit.client.0.smithi077.stdout:add osd2 bfa2bbb2-d3db-4b40-b51f-6142a5ebf004 2021-08-21T14:47:55.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:47:55.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAbEiFhuI1yJhAAdRSJLuefVbI79yU2cHnH+w== 2021-08-21T14:47:55.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAbEiFhuI1yJhAAdRSJLuefVbI79yU2cHnH+w=="}' 2021-08-21T14:47:55.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bfa2bbb2-d3db-4b40-b51f-6142a5ebf004 -i td/osd-backfill-recovery-log/2/new.json 2021-08-21T14:47:56.284 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:47:56.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-08-21T14:47:56.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAbEiFhuI1yJhAAdRSJLuefVbI79yU2cHnH+w== --osd-uuid bfa2bbb2-d3db-4b40-b51f-6142a5ebf004 2021-08-21T14:47:56.321 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.320+0000 7f7bc50f10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:56.323 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.322+0000 7f7bc50f10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:56.324 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.323+0000 7f7bc50f10c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:47:56.324 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.323+0000 7f7bc50f10c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T14:47:56.325 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.323+0000 7f7bc50f10c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T14:47:56.325 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.323+0000 7f7bc50f10c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T14:47:56.412 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:47:56.411+0000 7f7bc50f10c0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-08-21T14:48:02.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-08-21T14:48:02.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:48:02.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-08-21T14:48:02.199 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd2 key to auth repository 2021-08-21T14:48:02.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:48:02.702 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.2 2021-08-21T14:48:02.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-08-21T14:48:02.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:48:02.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:48:02.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:48:02.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:48:02.726 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:02.726+0000 7fd33a6060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:02.727 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:02.727+0000 7fd33a6060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:02.728 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:02.728+0000 7fd33a6060c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:03.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-08-21T14:48:03.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:48:03.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=2 2021-08-21T14:48:03.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:48:03.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:48:03.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:03.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:48:03.086 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:48:03.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:03.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:03.247 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:03.247+0000 7fd33a6060c0 -1 Falling back to public interface 2021-08-21T14:48:03.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:04.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:04.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:04.480 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:48:04.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:48:04.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:04.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:04.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:05.776 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:05.776+0000 7fd33a6060c0 -1 osd.2 0 log_to_monitors {default=true} 2021-08-21T14:48:05.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:05.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:05.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:48:05.924 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:48:05.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:05.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:06.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:07.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:07.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:07.505 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:48:07.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:48:07.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:07.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:08.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:09.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:09.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:09.026 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:48:09.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:48:09.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:09.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:09.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:10.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:10.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:10.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:48:10.411 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:48:10.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:10.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:10.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:11.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:11.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:11.925 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:48:11.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:48:11.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:11.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:12.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:13.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:13.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:13.331 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:48:13.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:48:13.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:13.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:13.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:14.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:14.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:14.884 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:48:14.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:48:14.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:14.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:15.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:16.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:16.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:16.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:48:16.271 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:48:16.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:16.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:16.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:17.730 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:17.730 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:17.731 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:48:17.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:48:17.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:17.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:18.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:19.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:19.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:19.275 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:48:19.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:48:19.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:19.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:19.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:20.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:20.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:20.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:48:20.985 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:48:20.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:20.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:21.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:22.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:22.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:22.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:48:22.576 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:48:22.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:22.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:23.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:24.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:24.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:24.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:48:24.302 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:48:24.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:24.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:25.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:26.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:26.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:26.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:48:26.119 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:48:26.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:26.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:27.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:28.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:28.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:28.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:48:28.121 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:48:28.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:28.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:48:28.548 INFO:tasks.workunit.client.0.smithi077.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58427,v1:127.0.0.1:6819/58427] [v2:127.0.0.1:6820/58427,v1:127.0.0.1:6821/58427] exists,up bfa2bbb2-d3db-4b40-b51f-6142a5ebf004 2021-08-21T14:48:28.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:48:28.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:48:28.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:48:28.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:48:28.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2021-08-21T14:48:28.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:48:28.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:48:28.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-08-21T14:48:28.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:48:28.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2021-08-21T14:48:28.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:48:28.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:48:28.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:48:28.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:48:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2021-08-21T14:48:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2021-08-21T14:48:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:48:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:48:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:48:28.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:48:28.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:48:28.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:48:28.557 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:48:28.558 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:48:28.558 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:48:28.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:48:28.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:48:28.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:48:28.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:48:28.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:48:28.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:48:28.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:48:28.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:48:28.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:48:28.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:48:28.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:48:28.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:48:28.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2021-08-21T14:48:28.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:48:28.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=ee7bea47-2e27-4f17-9317-4de306f83d16 2021-08-21T14:48:28.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 ee7bea47-2e27-4f17-9317-4de306f83d16' 2021-08-21T14:48:28.562 INFO:tasks.workunit.client.0.smithi077.stdout:add osd3 ee7bea47-2e27-4f17-9317-4de306f83d16 2021-08-21T14:48:28.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:48:28.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA8EiFhhkbQIRAAdnJtSbqwpGFJdZX2Cdzvag== 2021-08-21T14:48:28.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA8EiFhhkbQIRAAdnJtSbqwpGFJdZX2Cdzvag=="}' 2021-08-21T14:48:28.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new ee7bea47-2e27-4f17-9317-4de306f83d16 -i td/osd-backfill-recovery-log/3/new.json 2021-08-21T14:48:29.273 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:48:29.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2021-08-21T14:48:29.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA8EiFhhkbQIRAAdnJtSbqwpGFJdZX2Cdzvag== --osd-uuid ee7bea47-2e27-4f17-9317-4de306f83d16 2021-08-21T14:48:29.311 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.310+0000 7fb218f4f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:29.312 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.311+0000 7fb218f4f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:29.313 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.312+0000 7fb218f4f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:29.314 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.312+0000 7fb218f4f0c0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-08-21T14:48:29.314 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.312+0000 7fb218f4f0c0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-08-21T14:48:29.314 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.312+0000 7fb218f4f0c0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-08-21T14:48:29.388 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:29.388+0000 7fb218f4f0c0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2021-08-21T14:48:35.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2021-08-21T14:48:35.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:48:35.120 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd3 key to auth repository 2021-08-21T14:48:35.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-08-21T14:48:35.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:48:35.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-08-21T14:48:35.626 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.3 2021-08-21T14:48:35.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:48:35.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:48:35.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:48:35.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:48:35.649 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:35.648+0000 7fd1536440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:35.662 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:35.661+0000 7fd1536440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:35.663 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:35.662+0000 7fd1536440c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:48:36.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-08-21T14:48:36.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:48:36.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=3 2021-08-21T14:48:36.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:48:36.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:48:36.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:36.024 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:48:36.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:48:36.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:36.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:36.182 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:36.181+0000 7fd1536440c0 -1 Falling back to public interface 2021-08-21T14:48:36.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:37.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:37.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:37.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:48:37.456 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:48:37.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:37.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:37.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:38.611 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:48:38.610+0000 7fd1536440c0 -1 osd.3 0 log_to_monitors {default=true} 2021-08-21T14:48:38.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:38.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:38.827 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:48:38.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:48:38.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:38.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:39.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:40.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:40.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:40.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:48:40.447 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:48:40.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:40.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:40.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:41.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:41.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:41.886 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:48:41.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:48:41.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:41.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:42.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:43.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:43.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:43.377 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:48:43.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:48:43.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:43.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:43.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:44.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:44.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:44.814 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:48:44.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:48:44.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:44.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:45.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:46.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:46.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:46.238 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:48:46.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:48:46.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:46.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:46.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:47.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:47.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:47.704 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:48:47.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:48:47.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:47.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:48.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:49.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:49.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:49.153 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:48:49.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:48:49.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:49.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:49.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:50.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:50.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:50.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:48:50.581 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:48:50.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:50.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:50.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:51.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:51.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:51.986 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:48:51.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:48:51.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:51.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:52.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:53.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:53.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:53.476 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:48:53.476 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:48:53.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:53.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:53.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:54.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:54.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:54.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:48:54.859 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:48:54.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:54.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:55.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:56.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:56.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:56.267 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:48:56.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:48:56.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:56.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:57.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:48:58.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:48:58.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:48:58.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:48:58.065 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:48:58.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:48:58.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:48:59.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:00.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:00.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:00.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:49:00.081 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:49:00.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:00.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:49:00.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:01.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:01.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:01.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:49:01.664 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:49:01.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:01.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:49:02.141 INFO:tasks.workunit.client.0.smithi077.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/60177,v1:127.0.0.1:6827/60177] [v2:127.0.0.1:6828/60177,v1:127.0.0.1:6829/60177] exists,up ee7bea47-2e27-4f17-9317-4de306f83d16 2021-08-21T14:49:02.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:49:02.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:49:02.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:49:02.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:49:02.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2021-08-21T14:49:02.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:49:02.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:49:02.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-08-21T14:49:02.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:49:02.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2021-08-21T14:49:02.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:49:02.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:49:02.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:49:02.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:49:02.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2021-08-21T14:49:02.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2021-08-21T14:49:02.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:49:02.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:49:02.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:49:02.150 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:49:02.151 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:49:02.151 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:49:02.151 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:49:02.151 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:49:02.152 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:49:02.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:49:02.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:49:02.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:49:02.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:49:02.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:49:02.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:49:02.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:49:02.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:49:02.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:49:02.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:49:02.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:49:02.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:49:02.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2021-08-21T14:49:02.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:49:02.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=fa90f46c-05b1-4152-94d5-277d426786a1 2021-08-21T14:49:02.155 INFO:tasks.workunit.client.0.smithi077.stdout:add osd4 fa90f46c-05b1-4152-94d5-277d426786a1 2021-08-21T14:49:02.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 fa90f46c-05b1-4152-94d5-277d426786a1' 2021-08-21T14:49:02.156 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:49:02.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBeEiFhYE6TCRAAzoOTFxjsYqxoNeBJDacUmA== 2021-08-21T14:49:02.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBeEiFhYE6TCRAAzoOTFxjsYqxoNeBJDacUmA=="}' 2021-08-21T14:49:02.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new fa90f46c-05b1-4152-94d5-277d426786a1 -i td/osd-backfill-recovery-log/4/new.json 2021-08-21T14:49:02.840 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:49:02.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2021-08-21T14:49:02.849 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBeEiFhYE6TCRAAzoOTFxjsYqxoNeBJDacUmA== --osd-uuid fa90f46c-05b1-4152-94d5-277d426786a1 2021-08-21T14:49:02.874 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.873+0000 7f8f058c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:02.875 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.874+0000 7f8f058c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:02.876 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.875+0000 7f8f058c60c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:02.877 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.876+0000 7f8f058c60c0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-08-21T14:49:02.877 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.876+0000 7f8f058c60c0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-08-21T14:49:02.877 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.876+0000 7f8f058c60c0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-08-21T14:49:02.944 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:02.943+0000 7f8f058c60c0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2021-08-21T14:49:08.729 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2021-08-21T14:49:08.730 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:49:08.730 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd4 key to auth repository 2021-08-21T14:49:08.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2021-08-21T14:49:08.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:49:09.258 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.4 2021-08-21T14:49:09.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2021-08-21T14:49:09.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:49:09.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:49:09.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:49:09.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:49:09.281 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:09.280+0000 7fbbc14250c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:09.283 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:09.282+0000 7fbbc14250c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:09.284 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:09.283+0000 7fbbc14250c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:09.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2021-08-21T14:49:09.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:49:09.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=4 2021-08-21T14:49:09.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:49:09.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:49:09.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:09.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:49:09.660 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:49:09.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:09.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:09.801 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:09.800+0000 7fbbc14250c0 -1 Falling back to public interface 2021-08-21T14:49:10.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:11.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:11.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:11.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:49:11.035 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:49:11.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:11.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:11.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:12.252 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:12.251+0000 7fbbc14250c0 -1 osd.4 0 log_to_monitors {default=true} 2021-08-21T14:49:12.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:12.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:12.457 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:49:12.457 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:49:12.457 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:12.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:12.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:13.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:13.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:13.940 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:49:13.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:49:13.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:13.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:14.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:15.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:15.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:15.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:49:15.321 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:49:15.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:15.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:15.739 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:16.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:16.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:16.741 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:49:16.741 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:49:16.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:16.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:17.221 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:18.223 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:18.223 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:18.223 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:49:18.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:49:18.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:18.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:18.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:19.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:19.647 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:19.647 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:49:19.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:49:19.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:19.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:20.053 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:21.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:21.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:21.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:49:21.056 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:49:21.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:21.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:21.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:22.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:22.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:22.469 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:49:22.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:49:22.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:22.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:22.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:23.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:23.849 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:23.849 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:49:23.850 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:49:23.850 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:23.850 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:24.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:25.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:25.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:25.605 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:49:25.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:49:25.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:25.607 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:26.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:27.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:27.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:27.172 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:49:27.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:49:27.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:27.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:28.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:29.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:29.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:29.119 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:49:29.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:49:29.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:29.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:29.918 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:30.919 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:30.919 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:30.920 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:49:30.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:49:30.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:30.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:31.667 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:32.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:32.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:32.669 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:49:32.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:49:32.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:32.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:33.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:34.563 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:34.563 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:34.564 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:49:34.564 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:49:34.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:34.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:49:35.000 INFO:tasks.workunit.client.0.smithi077.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/61960,v1:127.0.0.1:6835/61960] [v2:127.0.0.1:6836/61960,v1:127.0.0.1:6837/61960] exists,up fa90f46c-05b1-4152-94d5-277d426786a1 2021-08-21T14:49:35.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:49:35.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:49:35.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:49:35.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:49:35.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2021-08-21T14:49:35.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:49:35.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:49:35.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-08-21T14:49:35.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:49:35.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2021-08-21T14:49:35.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:49:35.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:49:35.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:49:35.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:49:35.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2021-08-21T14:49:35.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2021-08-21T14:49:35.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:49:35.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2021-08-21T14:49:35.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:49:35.010 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:49:35.010 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:49:35.010 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:49:35.010 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:49:35.011 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:49:35.011 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:49:35.011 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:49:35.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:49:35.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:49:35.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:49:35.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:49:35.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:49:35.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:49:35.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:49:35.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:49:35.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:49:35.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:49:35.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:49:35.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2021-08-21T14:49:35.014 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:49:35.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bc56473b-c905-4bd9-85a0-1024030c51c5 2021-08-21T14:49:35.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd5 bc56473b-c905-4bd9-85a0-1024030c51c5' 2021-08-21T14:49:35.015 INFO:tasks.workunit.client.0.smithi077.stdout:add osd5 bc56473b-c905-4bd9-85a0-1024030c51c5 2021-08-21T14:49:35.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:49:35.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB/EiFh1EcnARAAyu5u7UmxLwlhi2Xmjbvd1A== 2021-08-21T14:49:35.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB/EiFh1EcnARAAyu5u7UmxLwlhi2Xmjbvd1A=="}' 2021-08-21T14:49:35.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bc56473b-c905-4bd9-85a0-1024030c51c5 -i td/osd-backfill-recovery-log/5/new.json 2021-08-21T14:49:35.519 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:49:35.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2021-08-21T14:49:35.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 5 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB/EiFh1EcnARAAyu5u7UmxLwlhi2Xmjbvd1A== --osd-uuid bc56473b-c905-4bd9-85a0-1024030c51c5 2021-08-21T14:49:35.556 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.555+0000 7fa96b7fb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:35.558 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.557+0000 7fa96b7fb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:35.559 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.558+0000 7fa96b7fb0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:35.559 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.558+0000 7fa96b7fb0c0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-08-21T14:49:35.560 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.558+0000 7fa96b7fb0c0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-08-21T14:49:35.560 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.558+0000 7fa96b7fb0c0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-08-21T14:49:35.609 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:35.609+0000 7fa96b7fb0c0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2021-08-21T14:49:41.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2021-08-21T14:49:41.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:49:41.602 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd5 key to auth repository 2021-08-21T14:49:41.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd5 key to auth repository 2021-08-21T14:49:41.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:49:42.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.5 2021-08-21T14:49:42.109 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.5 2021-08-21T14:49:42.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:49:42.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 5 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:49:42.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:49:42.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:49:42.132 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:42.132+0000 7f732f8ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:42.134 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:42.134+0000 7f732f8ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:42.135 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:42.135+0000 7f732f8ca0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:49:42.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 5 2021-08-21T14:49:42.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:49:42.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=5 2021-08-21T14:49:42.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:49:42.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:49:42.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:42.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:49:42.572 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:49:42.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:42.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:42.652 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:42.651+0000 7f732f8ca0c0 -1 Falling back to public interface 2021-08-21T14:49:43.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:44.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:44.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:44.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:49:44.006 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:49:44.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:44.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:44.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:45.155 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:49:45.154+0000 7f732f8ca0c0 -1 osd.5 0 log_to_monitors {default=true} 2021-08-21T14:49:45.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:45.388 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:45.388 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:49:45.388 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:49:45.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:45.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:45.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:46.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:46.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:46.967 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:49:46.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:49:46.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:46.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:47.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:48.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:48.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:48.390 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:49:48.390 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:49:48.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:48.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:48.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:49.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:49.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:49.933 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:49:49.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:49:49.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:49.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:50.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:51.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:51.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:51.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:49:51.327 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:49:51.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:51.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:51.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:52.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:52.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:52.807 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:49:52.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:49:52.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:52.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:53.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:54.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:54.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:54.193 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:49:54.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:49:54.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:54.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:54.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:55.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:55.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:55.575 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:49:55.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:49:55.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:55.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:55.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:56.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:56.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:56.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:49:56.957 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:49:56.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:56.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:57.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:49:58.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:49:58.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:49:58.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:49:58.662 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:49:58.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:49:58.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:49:59.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:50:00.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:50:00.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:50:00.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:50:00.285 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:50:00.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:50:00.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:50:00.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:50:01.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:50:01.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:50:01.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:50:01.699 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:50:01.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:50:01.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:50:02.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:50:03.186 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:50:03.186 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:50:03.187 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:50:03.187 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:50:03.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:50:03.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:50:03.816 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:50:04.817 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:50:04.818 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:50:04.818 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:50:04.818 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:50:04.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:50:04.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:50:05.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:50:06.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:50:06.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:50:06.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:50:06.692 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:50:06.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:50:06.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:50:07.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:50:08.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:50:08.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:50:08.366 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:50:08.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:50:08.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:50:08.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:50:08.795 INFO:tasks.workunit.client.0.smithi077.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/63709,v1:127.0.0.1:6843/63709] [v2:127.0.0.1:6844/63709,v1:127.0.0.1:6845/63709] exists,up bc56473b-c905-4bd9-85a0-1024030c51c5 2021-08-21T14:50:08.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:50:08.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:50:08.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:50:08.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2021-08-21T14:50:08.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-08-21T14:50:09.404 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test' already exists 2021-08-21T14:50:09.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:50:10.416 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2021-08-21T14:50:10.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:10.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2021-08-21T14:50:10.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:10.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2021-08-21T14:50:10.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:10.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2021-08-21T14:50:10.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:10.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2021-08-21T14:50:11.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2021-08-21T14:50:11.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2021-08-21T14:50:11.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2021-08-21T14:50:11.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2021-08-21T14:50:11.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2021-08-21T14:50:11.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2021-08-21T14:50:11.947 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:11.947 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2021-08-21T14:50:12.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2021-08-21T14:50:12.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2021-08-21T14:50:12.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.359 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2021-08-21T14:50:12.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2021-08-21T14:50:12.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.644 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2021-08-21T14:50:12.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2021-08-21T14:50:12.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:12.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2021-08-21T14:50:13.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2021-08-21T14:50:13.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.189 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2021-08-21T14:50:13.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2021-08-21T14:50:13.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2021-08-21T14:50:13.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2021-08-21T14:50:13.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2021-08-21T14:50:13.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2021-08-21T14:50:13.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:13.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2021-08-21T14:50:14.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2021-08-21T14:50:14.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2021-08-21T14:50:14.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2021-08-21T14:50:14.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2021-08-21T14:50:14.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2021-08-21T14:50:14.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2021-08-21T14:50:14.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2021-08-21T14:50:14.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:14.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2021-08-21T14:50:15.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2021-08-21T14:50:15.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2021-08-21T14:50:15.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2021-08-21T14:50:15.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2021-08-21T14:50:15.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2021-08-21T14:50:15.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2021-08-21T14:50:15.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:15.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2021-08-21T14:50:16.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2021-08-21T14:50:16.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2021-08-21T14:50:16.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2021-08-21T14:50:16.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2021-08-21T14:50:16.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2021-08-21T14:50:16.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2021-08-21T14:50:16.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2021-08-21T14:50:16.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:16.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2021-08-21T14:50:17.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:17.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2021-08-21T14:50:17.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:17.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2021-08-21T14:50:17.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:17.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2021-08-21T14:50:17.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:17.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2021-08-21T14:50:17.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:17.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2021-08-21T14:50:17.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:17.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2021-08-21T14:50:18.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2021-08-21T14:50:18.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2021-08-21T14:50:18.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2021-08-21T14:50:18.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.389 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2021-08-21T14:50:18.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2021-08-21T14:50:18.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2021-08-21T14:50:18.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2021-08-21T14:50:18.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2021-08-21T14:50:18.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:18.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2021-08-21T14:50:19.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2021-08-21T14:50:19.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2021-08-21T14:50:19.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2021-08-21T14:50:19.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2021-08-21T14:50:19.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2021-08-21T14:50:19.765 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.765 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2021-08-21T14:50:19.908 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:19.908 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2021-08-21T14:50:20.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2021-08-21T14:50:20.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2021-08-21T14:50:20.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2021-08-21T14:50:20.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2021-08-21T14:50:20.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2021-08-21T14:50:20.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2021-08-21T14:50:20.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2021-08-21T14:50:20.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:20.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2021-08-21T14:50:21.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2021-08-21T14:50:21.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2021-08-21T14:50:21.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2021-08-21T14:50:21.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2021-08-21T14:50:21.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2021-08-21T14:50:21.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2021-08-21T14:50:21.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2021-08-21T14:50:21.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:21.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2021-08-21T14:50:22.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2021-08-21T14:50:22.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2021-08-21T14:50:22.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2021-08-21T14:50:22.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.442 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2021-08-21T14:50:22.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2021-08-21T14:50:22.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2021-08-21T14:50:22.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2021-08-21T14:50:22.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:22.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2021-08-21T14:50:23.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2021-08-21T14:50:23.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.230 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2021-08-21T14:50:23.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2021-08-21T14:50:23.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2021-08-21T14:50:23.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2021-08-21T14:50:23.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2021-08-21T14:50:23.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:23.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2021-08-21T14:50:24.018 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2021-08-21T14:50:24.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2021-08-21T14:50:24.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2021-08-21T14:50:24.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2021-08-21T14:50:24.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2021-08-21T14:50:24.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2021-08-21T14:50:24.765 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.766 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2021-08-21T14:50:24.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:24.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2021-08-21T14:50:25.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2021-08-21T14:50:25.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2021-08-21T14:50:25.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2021-08-21T14:50:25.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.428 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2021-08-21T14:50:25.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2021-08-21T14:50:25.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2021-08-21T14:50:25.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2021-08-21T14:50:25.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:25.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2021-08-21T14:50:26.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2021-08-21T14:50:26.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2021-08-21T14:50:26.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2021-08-21T14:50:26.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2021-08-21T14:50:26.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2021-08-21T14:50:26.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2021-08-21T14:50:26.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2021-08-21T14:50:26.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:26.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2021-08-21T14:50:27.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2021-08-21T14:50:27.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2021-08-21T14:50:27.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2021-08-21T14:50:27.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2021-08-21T14:50:27.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2021-08-21T14:50:27.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2021-08-21T14:50:27.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:27.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2021-08-21T14:50:28.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2021-08-21T14:50:28.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2021-08-21T14:50:28.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2021-08-21T14:50:28.397 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.397 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2021-08-21T14:50:28.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2021-08-21T14:50:28.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2021-08-21T14:50:28.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2021-08-21T14:50:28.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2021-08-21T14:50:28.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:28.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2021-08-21T14:50:29.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:29.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2021-08-21T14:50:29.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:29.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2021-08-21T14:50:29.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:29.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2021-08-21T14:50:29.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:29.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2021-08-21T14:50:29.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:29.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2021-08-21T14:50:30.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:30.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2021-08-21T14:50:30.174 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:30.175 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2021-08-21T14:50:30.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:50:30.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2021-08-21T14:50:30.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2021-08-21T14:50:30.411 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2021-08-21T14:50:30.763 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:50:30.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2021-08-21T14:50:31.313 INFO:tasks.workunit.client.0.smithi077.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2021-08-21T14:50:31.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2021-08-21T14:50:31.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2021-08-21T14:50:32.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2021-08-21T14:50:32.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:50:32.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:50:32.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:50:32.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:50:32.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:50:32.326 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:50:32.326 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:50:32.327 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:50:32.327 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:50:32.327 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:50:32.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:50:32.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:50:32.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:50:32.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:50:32.428 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:50:32.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:50:32.800 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:50:32.800 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:50:32.800 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:50:32.801 INFO:tasks.workunit.client.0.smithi077.stderr:4 2021-08-21T14:50:32.801 INFO:tasks.workunit.client.0.smithi077.stderr:5' 2021-08-21T14:50:32.801 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:50:32.801 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:50:32.802 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:50:33.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836526 2021-08-21T14:50:33.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836526 2021-08-21T14:50:33.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526' 2021-08-21T14:50:33.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:50:33.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:50:33.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673000 2021-08-21T14:50:33.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673000 2021-08-21T14:50:33.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673000' 2021-08-21T14:50:33.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:50:33.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:50:33.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509477 2021-08-21T14:50:33.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509477 2021-08-21T14:50:33.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509477' 2021-08-21T14:50:33.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:50:33.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:50:33.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345951 2021-08-21T14:50:33.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345951 2021-08-21T14:50:33.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509477 3-85899345951' 2021-08-21T14:50:33.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:50:33.780 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:50:33.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182424 2021-08-21T14:50:33.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182424 2021-08-21T14:50:33.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509477 3-85899345951 4-107374182424' 2021-08-21T14:50:33.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:50:33.913 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-08-21T14:50:34.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=128849018894 2021-08-21T14:50:34.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 128849018894 2021-08-21T14:50:34.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509477 3-85899345951 4-107374182424 5-128849018894' 2021-08-21T14:50:34.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:50:34.046 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836526 2021-08-21T14:50:34.046 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:50:34.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:50:34.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836526 2021-08-21T14:50:34.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:50:34.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836526 2021-08-21T14:50:34.049 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836526 2021-08-21T14:50:34.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2021-08-21T14:50:34.049 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:50:34.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836524 -lt 21474836526 2021-08-21T14:50:34.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:50:35.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:50:35.431 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:50:35.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836526 -lt 21474836526 2021-08-21T14:50:35.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:50:35.852 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673000 2021-08-21T14:50:35.852 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:50:35.853 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:50:35.853 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673000 2021-08-21T14:50:35.854 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:50:35.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673000 2021-08-21T14:50:35.855 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673000 2021-08-21T14:50:35.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2021-08-21T14:50:35.856 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:50:36.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673000 -lt 42949673000 2021-08-21T14:50:36.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:50:36.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509477 2021-08-21T14:50:36.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:50:36.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:50:36.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509477 2021-08-21T14:50:36.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:50:36.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509477 2021-08-21T14:50:36.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2021-08-21T14:50:36.315 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509477 2021-08-21T14:50:36.315 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:50:36.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509478 -lt 64424509477 2021-08-21T14:50:36.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:50:36.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345951 2021-08-21T14:50:36.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:50:36.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:50:36.789 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345951 2021-08-21T14:50:36.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:50:36.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345951 2021-08-21T14:50:36.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2021-08-21T14:50:36.791 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345951 2021-08-21T14:50:36.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:50:37.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345951 -lt 85899345951 2021-08-21T14:50:37.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:50:37.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182424 2021-08-21T14:50:37.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:50:37.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:50:37.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182424 2021-08-21T14:50:37.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:50:37.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182424 2021-08-21T14:50:37.208 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182424 2021-08-21T14:50:37.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182424' 2021-08-21T14:50:37.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:50:37.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182425 -lt 107374182424 2021-08-21T14:50:37.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:50:37.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 5-128849018894 2021-08-21T14:50:37.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:50:37.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=5 2021-08-21T14:50:37.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 5-128849018894 2021-08-21T14:50:37.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:50:37.604 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.5 seq 128849018894 2021-08-21T14:50:37.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=128849018894 2021-08-21T14:50:37.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2021-08-21T14:50:37.605 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 5 2021-08-21T14:50:37.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 128849018894 -lt 128849018894 2021-08-21T14:50:37.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:50:37.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:37.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:38.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 1 == 0 2021-08-21T14:50:38.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:38.529 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:38.530 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:38.530 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:38.530 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:38.530 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:38.531 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:38.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:38.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:38.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:38.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:39.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:39.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' -1 2021-08-21T14:50:39.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: wait_for_clean: loop=0 2021-08-21T14:50:39.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: num_active_clean=0 2021-08-21T14:50:39.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:39.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:39.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:39.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:39.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:39.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:39.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:39.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:39.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:39.565 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:39.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:39.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:39.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:39.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:40.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:40.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:40.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:40.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:40.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:40.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:40.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:40.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:40.435 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:40.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=1389 2021-08-21T14:50:40.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 1389 '!=' null 2021-08-21T14:50:40.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:40.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:40.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:41.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:41.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:41.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:41.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:41.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:41.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:41.067 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:41.067 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:41.433 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:41.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:41.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:41.435 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:42.001 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:42.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:42.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:42.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:42.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:42.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:42.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:42.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:42.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:42.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=1114 2021-08-21T14:50:42.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 1114 '!=' null 2021-08-21T14:50:42.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:42.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:42.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:42.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:42.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:42.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:42.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:42.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:42.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:42.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:42.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:43.001 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:43.002 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:43.003 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:43.003 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:43.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:43.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:43.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:43.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:43.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:43.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:43.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:43.498 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:43.499 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:44.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:50:44.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:50:44.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:44.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:44.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:44.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:44.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:44.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:44.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:44.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:44.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:44.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:44.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:44.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:44.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:44.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:44.509 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:45.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:45.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:45.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:45.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:45.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:45.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:45.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:45.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:45.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:45.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3887 2021-08-21T14:50:45.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3887 '!=' null 2021-08-21T14:50:45.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:45.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:45.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:45.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:45.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:45.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:45.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:45.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:45.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:45.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:45.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:45.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:45.988 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:45.989 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:45.989 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:46.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:46.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:46.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:46.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:46.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:46.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:46.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:46.485 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:46.485 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:47.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3887 2021-08-21T14:50:47.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3887 '!=' null 2021-08-21T14:50:47.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:47.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:47.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:47.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:47.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:47.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:47.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:47.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:47.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:47.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:47.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:47.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:47.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:47.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:47.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:48.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:48.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:48.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:48.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:48.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:48.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:48.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:48.065 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:48.065 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:48.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:50:48.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:50:48.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:48.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:48.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:48.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:48.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:48.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:48.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:48.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:48.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:48.675 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:48.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:49.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:49.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:49.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:49.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:49.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:49.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:49.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:49.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:49.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:49.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:49.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:49.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:49.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:50:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:50:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:50.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:50.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:50.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:50.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:50.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:50.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:50.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:50.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:50.207 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:50.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:50.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:50.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:50.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:51.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:51.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:51.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:51.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:51.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:51.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:51.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:51.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:51.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:51.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:50:51.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:50:51.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:51.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:51.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:51.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:51.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:51.714 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:51.715 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:51.715 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:51.715 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:51.715 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:51.716 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:52.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:52.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:52.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:52.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:52.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:52.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:52.614 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:52.615 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:52.615 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:52.615 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:52.615 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:52.616 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:52.616 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:53.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:50:53.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:50:53.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:53.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:53.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:53.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:53.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:53.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:53.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:53.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:53.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:53.248 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:53.248 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:53.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:53.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:53.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:53.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:54.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:54.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:54.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:54.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:54.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:54.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:54.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:54.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:54.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:54.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:50:54.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:50:54.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:54.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:54.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:54.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:54.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:54.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:54.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:54.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:54.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:54.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:54.817 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:55.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:55.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:55.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:55.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:55.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:55.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:55.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:55.678 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:55.679 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:56.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:50:56.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:50:56.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:56.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:56.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:56.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:56.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:56.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:56.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:56.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:56.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:56.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:56.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:56.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:56.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:56.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:56.646 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:57.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:57.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:57.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:57.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:57.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:57.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:57.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:57.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:57.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:57.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:50:57.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:50:57.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:57.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:57.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:57.889 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:57.889 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:57.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:57.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:57.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:57.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:57.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:57.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:58.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:58.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:58.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:58.263 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:50:58.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:50:58.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:50:58.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:50:58.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:50:58.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:50:58.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:50:58.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:50:58.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:50:58.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:50:59.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:50:59.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:50:59.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:50:59.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:50:59.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:50:59.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:50:59.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:50:59.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:50:59.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:50:59.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:50:59.426 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:50:59.426 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:50:59.426 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:50:59.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:50:59.796 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:50:59.796 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:50:59.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:00.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:00.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:00.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:00.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:00.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:00.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:00.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:00.303 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:00.303 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:00.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:51:00.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:51:00.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:00.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:00.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:00.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:00.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:00.967 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:00.967 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:00.967 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:00.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:00.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:00.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:01.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:01.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:01.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:01.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:01.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:01.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:01.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:01.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:01.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:01.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:01.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:01.843 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:01.843 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:02.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:51:02.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:51:02.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:02.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:02.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:02.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:02.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:02.468 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:02.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:02.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:02.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:02.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:02.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:02.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:02.833 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:02.833 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:02.833 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:03.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:03.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:03.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:03.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:03.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:03.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:03.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:03.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:03.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:03.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:03.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:03.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:03.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:03.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:03.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:03.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:03.940 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:03.940 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:03.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:03.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:03.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:03.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:04.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:04.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:04.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:04.303 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:04.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:04.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:04.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:04.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:04.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:04.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:04.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:04.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:04.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:05.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:05.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:05.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:05.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:05.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:05.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:05.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:05.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:05.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:05.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:05.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:05.464 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:05.464 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:05.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:05.870 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:05.870 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:05.870 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:06.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:06.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:06.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:06.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:06.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:06.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:06.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:06.374 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:06.374 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:06.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:06.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:06.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:06.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:06.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:06.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:06.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:07.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:07.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:07.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:07.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:07.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:07.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:07.370 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:07.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:07.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:07.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:07.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:07.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:07.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:07.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:07.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:07.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:07.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:07.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:07.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:08.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:08.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:08.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:08.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:08.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:08.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:08.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:08.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:08.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:08.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:08.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:08.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:08.563 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:08.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:08.939 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:08.940 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:08.940 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:09.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:09.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:09.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:09.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:09.463 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:09.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:09.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:09.464 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:09.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:10.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:10.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:10.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:10.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:10.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:10.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:10.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:10.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:10.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:10.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:10.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:10.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:10.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:10.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:10.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:10.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:10.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:11.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:11.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:11.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:11.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:11.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:11.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:11.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:11.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:11.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:11.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:11.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:11.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:11.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:11.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:11.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:11.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:11.628 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:11.628 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:11.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:11.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:11.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:11.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:11.994 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:11.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:11.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:11.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:12.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:12.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:12.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:12.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:12.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:12.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:12.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:12.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:12.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:13.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:13.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:13.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:13.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:13.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:13.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:13.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:13.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:13.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:13.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:13.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:13.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:13.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:13.607 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:13.608 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:13.608 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:13.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:14.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:14.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:14.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:14.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:14.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:14.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:14.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:14.128 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:14.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:14.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:51:14.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:51:14.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:14.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:14.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:14.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:14.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:14.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:14.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:14.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:14.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:14.772 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:14.772 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:15.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:15.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:15.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:15.156 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:15.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:15.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:15.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:15.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:15.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:15.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:15.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:15.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:15.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:16.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:16.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:16.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:16.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:16.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:16.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:16.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:16.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:16.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:16.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:16.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:16.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:16.295 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:16.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:16.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:16.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:16.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:17.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:17.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:17.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:17.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:17.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:17.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:17.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:17.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:17.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:17.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:17.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:17.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:17.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:17.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:17.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:17.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:17.845 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:17.845 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:17.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:17.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:17.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:17.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:18.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:18.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:18.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:18.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:18.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:18.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:18.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:18.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:18.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:18.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:18.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:18.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:18.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:19.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:19.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:19.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:19.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:19.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:19.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:19.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:19.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:19.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:19.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:19.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:19.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:19.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:19.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:19.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:19.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:19.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:20.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:20.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:20.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:20.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:20.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:20.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:20.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:20.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:20.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:20.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:20.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:20.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:20.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:20.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:20.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:20.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:20.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:20.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:20.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:20.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:20.971 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:20.971 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:21.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:21.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:21.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:21.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:21.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:21.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:21.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:21.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:21.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:21.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:21.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:21.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:21.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:22.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:22.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:22.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:22.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:22.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:22.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:22.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:22.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:22.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:22.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:22.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:22.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:22.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:22.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:22.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:22.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:22.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:23.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:23.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:23.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:23.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:23.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:23.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:23.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:23.383 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:23.383 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:23.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:23.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:23.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:23.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:23.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:24.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:24.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:24.003 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:24.003 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:24.003 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:24.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:24.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:24.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:24.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:24.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:24.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:24.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:24.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:24.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:24.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:24.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:24.882 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:24.882 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:25.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:51:25.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:51:25.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:25.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:25.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:25.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:25.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:25.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:25.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:25.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:25.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:25.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:25.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:25.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:25.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:25.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:25.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:26.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:26.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:26.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:26.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:26.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:26.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:26.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:26.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:26.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:26.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:51:26.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:51:26.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:26.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:26.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:27.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:27.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:27.016 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:27.016 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:27.016 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:27.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:27.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:27.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:27.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:27.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:27.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:27.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:27.908 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:27.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:27.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:27.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:27.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:27.910 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:27.910 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:27.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:27.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:28.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:51:28.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:51:28.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:28.411 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:28.411 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:28.510 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:28.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:28.512 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:28.512 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:28.512 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:28.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:28.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:28.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:28.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:28.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:28.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:28.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:29.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:29.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:29.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:29.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:29.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:29.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:29.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:29.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:29.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:29.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:29.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:29.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:29.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:29.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:30.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:30.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:30.018 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:30.019 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:30.019 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:30.019 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:30.019 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:30.020 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:30.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:30.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:30.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:30.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:30.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:30.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:30.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:30.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:30.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:30.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:30.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:30.906 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:30.906 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:31.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:31.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:31.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:31.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:31.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:31.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:31.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:31.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:31.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:31.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:31.568 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:31.568 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:31.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:31.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:31.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:31.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:32.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:32.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:32.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:32.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:32.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:32.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:32.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:32.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:32.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:32.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:32.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:32.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:32.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:32.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:33.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:33.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:33.065 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:33.065 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:33.065 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:33.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:33.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:33.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:33.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:33.435 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:33.435 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:33.436 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:33.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:33.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:33.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:33.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:33.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:33.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:33.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:33.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:33.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:34.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:34.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:34.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:34.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:34.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:34.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:34.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:34.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:34.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:34.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:34.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:34.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:34.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:34.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:34.994 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:34.994 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:34.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:35.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:35.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:35.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:35.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:35.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:35.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:35.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:35.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:35.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:36.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:36.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:36.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:36.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:36.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:36.174 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:36.174 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:36.175 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:36.175 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:36.176 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:36.176 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:36.176 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:36.176 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:36.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:36.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:36.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:36.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:37.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:37.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:37.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:37.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:37.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:37.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:37.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:37.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:37.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:37.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:37.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:37.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:37.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:37.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:37.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:37.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:37.652 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:37.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:37.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:37.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:37.653 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:37.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:38.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:38.023 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:38.024 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:38.024 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:38.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:38.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:38.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:38.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:38.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:38.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:38.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:38.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:38.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:39.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:39.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:39.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:39.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:39.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:39.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:39.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:39.156 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:39.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:39.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:39.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:39.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:39.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:40.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:40.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:40.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:40.022 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:40.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:40.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:40.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:40.024 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:40.024 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:40.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:40.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:40.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:40.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:40.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:40.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:40.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:40.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:40.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:40.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:40.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:40.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:40.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:41.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:41.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:41.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:41.008 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:41.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:41.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:41.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:41.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:41.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:41.510 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:41.510 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:41.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:41.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:42.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:51:42.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:51:42.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:42.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:42.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:42.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:42.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:42.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:42.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:42.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:42.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:42.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:42.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:42.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:42.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:42.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:42.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:43.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:43.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:43.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:43.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:43.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:43.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:43.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:43.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:43.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:43.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:43.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:43.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:43.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:43.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:43.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:43.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:43.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:43.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:43.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:43.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:43.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:43.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:44.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:44.146 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:44.146 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:44.146 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:44.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:44.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:44.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:44.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:44.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:44.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:44.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:44.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:44.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:45.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:51:45.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:51:45.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:45.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:45.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:45.247 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:45.247 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:45.248 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:45.249 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:45.249 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:45.249 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:45.249 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:45.250 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:45.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:45.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:45.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:45.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:46.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:46.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:46.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:46.615 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:46.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:46.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:46.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:46.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:46.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:46.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:46.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:46.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:46.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:46.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:46.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:46.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:47.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:47.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:47.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:47.117 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:47.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:47.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:47.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:47.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:47.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:47.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:47.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:47.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:47.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:48.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:48.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:48.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:48.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:48.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:48.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:48.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:48.260 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:48.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:48.632 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:48.632 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:48.632 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:49.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:49.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:49.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:49.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:49.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:49.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:49.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:49.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:49.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:49.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:51:49.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:51:49.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:49.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:49.763 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:49.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:49.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:49.864 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:49.864 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:49.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:49.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:49.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:49.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:50.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:50.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:50.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:50.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:50.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:50.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:50.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:50.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:50.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:50.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:50.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:50.736 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:50.736 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:51.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:51:51.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:51:51.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:51.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:51.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:51.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:51.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:51.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:51.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:51.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:51.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:51.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:51.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:51.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:51.755 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:51.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:51.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:52.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:52.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:52.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:52.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:52.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:52.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:52.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:52.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:52.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:52.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:51:52.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:51:52.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:52.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:52.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:52.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:52.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:52.900 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:52.901 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:52.901 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:52.901 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:52.901 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:52.902 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:53.281 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:53.283 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:53.283 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:53.284 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:53.784 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:53.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:53.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:53.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:53.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:53.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:53.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:53.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:53.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:54.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:51:54.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:51:54.282 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:54.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:54.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:54.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:54.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:54.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:54.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:54.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:54.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:54.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:54.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:54.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:54.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:54.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:54.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:55.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:55.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:55.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:55.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:55.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:55.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:55.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:55.264 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:55.264 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:55.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:55.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:55.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:55.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:55.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:55.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:55.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:55.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:55.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:55.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:55.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:55.899 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:55.899 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:56.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:56.268 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:56.269 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:56.269 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:56.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:56.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:56.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:56.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:56.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:56.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:56.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:56.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:56.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:57.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:57.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:57.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:57.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:57.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:57.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:57.410 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:57.411 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:57.411 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:57.411 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:57.412 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:57.412 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:57.412 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:57.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:57.822 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:57.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:57.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:58.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:58.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:58.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:58.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:58.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:58.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:58.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:58.332 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:58.333 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:51:58.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:51:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:51:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:51:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:51:58.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:51:58.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:51:58.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:51:58.944 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:51:58.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:51:58.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:51:58.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:51:58.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:51:58.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:51:59.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:51:59.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:51:59.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:51:59.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:51:59.911 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:51:59.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:51:59.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:51:59.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:51:59.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:51:59.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:51:59.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:51:59.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:51:59.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:00.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:52:00.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:52:00.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:00.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:00.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:00.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:00.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:00.560 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:00.560 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:00.560 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:00.560 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:00.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:00.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:00.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:00.932 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:00.932 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:00.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:01.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:01.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:01.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:01.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:01.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:01.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:01.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:01.467 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:01.467 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:02.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:02.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:02.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:02.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:02.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:02.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:02.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:02.143 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:02.143 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:02.143 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:02.143 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:02.143 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:02.144 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:02.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:02.509 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:02.509 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:02.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:03.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:03.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:03.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:03.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:03.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:03.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:03.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:03.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:03.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:03.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:03.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:03.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:03.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:03.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:03.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:03.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:03.632 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:03.632 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:03.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:03.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:03.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:03.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:04.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:04.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:04.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:04.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:04.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:04.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:04.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:04.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:04.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:04.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:04.509 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:04.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:04.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:05.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:05.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:05.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:05.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:05.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:05.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:05.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:05.114 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:05.114 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:05.114 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:05.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:05.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:05.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:05.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:05.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:05.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:05.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:05.983 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:05.983 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:05.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:05.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:05.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:05.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:05.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:05.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:05.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:06.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:06.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:06.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:06.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:06.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:06.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:06.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:06.615 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:06.615 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:06.615 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:06.616 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:06.616 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:06.616 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:06.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:06.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:06.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:06.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:07.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:07.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:07.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:07.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:07.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:07.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:07.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:07.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:07.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:08.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:08.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:08.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:08.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:08.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:08.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:08.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:08.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:08.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:08.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:08.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:08.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:08.163 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:08.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:08.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:08.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:08.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:09.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:09.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:09.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:09.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:09.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:09.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:09.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:09.039 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:09.039 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:09.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:09.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:09.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:09.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:09.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:09.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:09.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:09.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:09.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:09.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:09.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:09.650 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:09.651 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:10.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:10.020 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:10.021 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:10.021 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:10.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:10.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:10.525 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:10.525 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:11.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:11.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:11.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:11.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:11.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:11.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:11.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:11.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:11.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:11.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:11.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:11.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:11.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:11.598 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:11.600 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:11.600 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:11.600 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:12.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:12.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:12.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:12.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:12.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:12.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:12.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:12.102 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:12.102 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:12.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:52:12.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:52:12.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:12.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:12.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:12.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:12.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:12.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:12.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:12.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:12.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:12.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:12.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:13.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:13.102 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:13.102 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:13.102 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:13.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:13.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:13.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:13.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:13.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:13.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:13.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:13.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:13.613 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:14.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:14.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:14.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:14.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:14.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:14.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:14.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:14.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:14.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:14.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:14.215 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:14.215 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:14.215 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:14.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:14.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:14.578 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:14.578 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:15.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:15.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:15.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:15.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:15.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:15.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:15.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:15.154 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:15.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:15.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:15.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:15.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:15.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:15.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:15.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:15.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:15.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:15.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:15.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:15.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:15.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:15.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:16.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:16.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:16.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:16.187 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:16.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:16.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:16.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:16.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:16.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:16.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:16.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:16.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:16.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:17.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:17.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:17.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:17.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:17.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:17.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:17.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:17.335 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:17.335 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:17.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:17.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:17.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:17.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:17.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:17.713 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:17.713 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:17.713 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:18.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:18.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:18.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:18.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:18.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:18.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:18.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:18.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:18.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:18.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:18.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:18.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:18.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:18.705 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:18.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:18.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:18.806 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:18.806 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:18.807 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:18.807 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:18.807 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:18.807 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:19.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:19.171 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:19.171 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:19.172 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:19.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:19.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:19.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:19.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:19.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:19.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:19.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:19.704 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:19.704 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:20.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:20.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:20.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:20.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:20.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:20.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:20.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:20.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:20.307 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:20.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:20.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:20.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:20.665 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:21.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:21.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:21.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:21.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:21.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:21.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:21.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:21.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:21.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:21.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:21.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:21.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:21.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:21.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:21.801 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:21.801 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:21.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:21.802 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:21.802 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:21.803 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:21.803 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:21.803 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:22.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:22.168 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:22.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:22.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:22.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:22.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:22.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:22.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:22.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:22.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:22.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:22.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:22.697 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:23.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:52:23.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:52:23.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:23.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:23.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:23.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:23.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:23.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:23.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:23.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:23.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:23.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:23.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:23.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:23.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:23.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:23.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:24.254 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:24.255 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:24.255 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:24.255 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:24.256 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:24.256 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:24.256 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:24.256 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:24.256 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:24.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:52:24.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:52:24.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:24.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:24.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:24.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:24.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:24.924 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:24.925 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:24.925 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:24.925 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:24.926 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:24.926 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:25.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:25.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:25.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:25.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:25.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:25.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:25.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:25.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:25.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:25.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:25.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:25.805 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:25.805 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:26.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:26.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:26.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:26.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:26.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:26.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:26.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:26.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:26.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:26.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:26.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:26.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:26.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:26.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:26.843 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:26.843 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:26.844 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:27.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:27.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:27.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:27.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:27.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:27.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:27.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:27.349 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:27.349 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:27.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:27.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:27.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:27.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:27.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:27.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:27.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:27.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:27.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:27.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:27.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:27.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:27.961 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:28.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:28.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:28.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:28.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:28.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:28.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:28.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:28.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:28.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:28.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:28.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:28.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:28.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:29.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:29.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:29.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:29.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:29.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:29.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:29.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:29.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:29.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:29.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:29.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:29.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:29.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:29.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:29.893 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:29.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:29.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:30.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:30.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:30.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:30.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:30.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:30.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:30.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:30.405 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:30.405 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:30.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:30.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:30.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:30.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:30.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:30.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:30.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:30.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:30.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:30.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:30.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:30.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:30.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:31.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:31.366 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:31.366 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:31.367 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:31.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:31.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:31.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:31.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:31.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:31.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:31.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:31.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:31.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:32.396 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:32.396 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:32.397 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:32.397 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:32.397 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:32.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:32.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:32.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:32.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:32.498 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:32.498 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:32.498 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:32.498 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:32.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:32.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:32.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:32.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:33.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:33.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:33.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:33.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:33.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:33.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:33.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:33.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:33.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:33.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:33.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:33.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:33.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:33.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:34.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:34.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:34.087 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:34.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:34.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:34.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:34.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:34.088 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:34.457 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:34.458 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:34.458 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:34.459 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:35.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:35.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:35.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:35.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:35.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:35.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:35.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:35.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:35.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:35.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:52:35.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:52:35.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:35.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:35.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:35.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:35.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:35.751 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:35.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:35.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:35.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:35.752 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:35.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:36.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:36.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:36.123 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:36.123 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:36.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:36.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:36.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:36.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:36.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:37.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:52:37.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:52:37.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:37.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:37.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:37.254 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:37.255 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:37.256 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:37.256 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:37.256 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:37.256 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:37.257 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:37.257 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:37.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:37.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:37.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:37.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:38.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:38.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:38.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:38.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:38.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:38.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:38.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:38.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:38.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:38.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:52:38.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:52:38.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:38.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:38.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:38.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:38.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:38.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:38.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:38.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:38.757 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:38.757 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:38.757 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:39.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:39.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:39.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:39.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:39.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:39.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:39.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:39.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:39.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:39.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:40.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:40.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:40.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:40.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:40.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:40.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:40.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:40.267 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:40.268 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:40.268 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:40.268 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:40.268 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:40.269 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:40.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:40.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:40.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:40.632 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:41.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:41.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:41.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:41.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:41.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:41.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:41.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:41.121 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:41.121 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:41.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:41.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:41.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:41.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:41.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:41.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:41.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:41.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:41.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:41.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:41.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:41.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:41.772 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:42.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:42.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:42.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:42.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:42.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:42.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:42.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:42.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:42.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:42.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:42.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:42.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:42.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:43.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:43.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:43.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:43.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:43.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:43.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:43.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:43.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:43.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:43.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:43.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:43.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:43.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:43.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:43.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:43.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:43.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:44.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:44.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:44.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:44.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:44.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:44.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:44.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:44.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:44.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:44.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:44.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:44.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:44.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:44.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:44.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:44.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:44.813 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:44.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:44.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:44.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:44.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:44.815 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:45.185 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:45.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:45.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:45.187 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:45.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:45.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:45.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:45.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:45.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:45.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:45.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:45.701 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:45.701 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:46.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:46.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:46.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:46.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:46.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:46.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:46.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:46.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:46.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:46.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:46.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:46.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:46.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:46.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:46.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:46.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:46.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:47.248 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:47.248 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:47.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:47.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:47.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:47.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:47.249 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:47.250 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:47.250 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:47.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:47.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:47.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:47.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:47.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:47.870 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:47.870 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:47.871 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:47.871 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:47.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:47.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:47.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:47.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:48.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:48.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:48.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:48.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:48.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:48.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:48.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:48.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:48.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:48.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:48.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:48.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:48.747 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:49.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:49.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:49.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:49.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:49.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:49.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:49.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:49.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:49.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:49.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:49.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:49.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:49.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:49.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:49.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:49.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:49.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:50.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:50.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:50.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:50.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:50.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:50.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:50.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:50.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:50.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:50.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:50.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:50.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:50.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:50.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:50.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:50.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:50.906 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:50.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:50.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:50.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:50.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:50.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:51.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:51.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:51.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:51.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:51.801 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:51.802 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:51.802 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:51.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:51.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:51.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:51.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:51.804 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:51.804 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:52.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:52:52.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:52:52.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:52.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:52.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:52.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:52.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:52.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:52.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:52.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:52.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:52.405 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:52.405 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:52.774 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:52.775 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:52.775 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:52.775 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:53.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:53.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:53.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:53.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:53.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:53.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:53.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:53.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:53.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:53.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:52:53.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:52:53.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:53.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:53.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:53.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:53.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:53.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:53.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:53.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:53.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:53.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:53.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:54.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:54.329 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:54.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:54.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:54.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:54.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:54.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:54.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:54.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:54.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:54.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:54.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:54.846 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:55.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:55.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:55.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:55.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:55.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:55.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:55.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:55.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:55.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:55.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:55.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:55.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:55.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:55.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:55.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:55.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:55.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:56.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:56.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:56.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:56.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:56.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:56.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:56.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:56.374 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:56.374 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:56.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:56.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:56.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:56.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:56.925 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:57.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:57.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:57.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:57.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:57.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:57.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:57.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:57.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:57.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:57.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:57.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:57.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:57.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:57.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:57.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:57.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:57.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:57.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:57.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:57.897 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:57.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:58.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:52:58.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:52:58.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:58.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:58.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:52:58.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:52:58.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:52:58.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:52:58.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:52:58.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:52:58.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:52:58.524 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:52:58.524 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:52:58.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:52:58.903 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:52:58.903 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:52:58.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:52:59.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:52:59.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:52:59.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:52:59.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:52:59.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:52:59.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:52:59.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:52:59.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:52:59.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:52:59.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:52:59.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:52:59.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:52:59.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:52:59.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:00.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:00.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:00.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:00.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:00.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:00.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:00.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:00.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:00.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:00.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:00.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:00.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:00.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:00.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:00.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:00.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:00.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:00.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:00.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:00.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:00.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:01.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:53:01.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:53:01.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:01.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:01.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:01.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:01.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:01.522 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:01.522 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:01.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:01.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:01.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:01.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:01.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:01.885 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:01.885 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:01.886 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:02.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:02.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:02.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:02.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:02.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:02.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:02.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:02.380 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:02.380 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:02.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:53:02.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:53:02.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:02.898 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:02.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:02.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:02.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:03.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:03.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:03.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:03.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:03.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:03.002 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:03.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:03.366 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:03.367 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:03.367 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:03.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:03.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:03.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:03.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:03.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:03.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:03.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:03.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:03.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:04.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:53:04.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:53:04.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:04.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:04.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:04.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:04.481 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:04.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:04.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:04.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:04.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:04.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:04.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:04.849 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:04.850 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:04.851 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:04.851 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:05.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:05.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:05.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:05.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:05.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:05.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:05.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:05.352 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:05.353 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:05.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:05.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:05.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:05.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:05.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:05.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:05.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:05.973 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:05.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:05.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:05.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:05.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:05.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:06.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:06.335 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:06.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:06.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:06.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:06.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:06.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:06.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:06.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:06.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:06.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:06.858 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:06.858 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:07.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:07.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:07.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:07.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:07.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:07.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:07.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:07.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:07.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:07.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:07.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:07.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:07.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:07.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:07.887 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:07.887 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:07.887 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:08.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:08.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:08.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:08.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:08.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:08.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:08.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:08.402 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:08.402 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:08.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:08.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:08.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:08.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:08.994 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:09.093 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:09.093 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:09.094 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:09.095 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:09.095 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:09.095 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:09.096 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:09.096 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:09.462 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:09.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:09.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:09.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:10.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:10.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:10.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:10.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:10.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:10.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:10.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:10.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:10.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:10.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:53:10.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:53:10.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:10.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:10.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:10.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:10.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:10.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:10.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:10.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:10.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:10.688 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:10.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:11.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:11.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:11.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:11.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:11.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:11.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:11.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:11.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:11.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:11.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:11.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:11.573 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:11.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:12.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:12.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:12.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:12.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:12.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:12.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:12.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:12.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:12.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:12.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:12.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:12.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:12.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:12.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:12.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:12.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:12.553 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:13.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:13.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:13.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:13.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:13.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:13.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:13.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:13.063 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:13.063 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:13.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:13.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:13.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:13.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:13.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:13.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:13.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:13.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:13.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:13.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:13.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:13.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:13.762 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:14.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:14.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:14.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:14.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:14.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:14.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:14.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:14.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:14.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:14.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:14.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:14.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:14.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:15.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:15.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:15.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:15.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:15.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:15.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:15.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:15.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:15.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:15.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:15.238 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:15.238 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:15.238 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:15.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:15.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:15.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:15.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:16.096 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:16.096 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:16.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:16.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:16.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:16.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:16.097 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:16.098 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:16.098 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:16.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:16.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:16.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:16.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:16.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:16.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:16.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:16.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:16.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:16.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:16.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:16.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:16.735 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:17.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:17.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:17.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:17.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:17.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:17.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:17.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:17.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:17.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:17.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:17.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:17.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:17.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:18.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:18.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:18.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:18.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:18.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:18.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:18.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:18.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:18.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:18.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:18.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:18.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:18.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:18.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:18.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:18.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:18.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:19.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:19.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:19.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:19.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:19.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:19.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:19.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:19.141 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:19.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:19.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:19.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:19.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:19.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:19.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:19.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:19.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:19.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:19.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:19.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:19.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:19.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:19.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:20.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:20.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:20.158 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:20.158 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:20.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:20.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:20.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:20.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:20.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:20.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:20.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:20.722 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:20.723 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:21.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:53:21.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:53:21.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:21.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:21.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:21.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:21.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:21.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:21.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:21.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:21.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:21.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:21.467 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:21.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:21.832 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:21.833 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:21.833 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:22.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:22.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:22.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:22.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:22.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:22.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:22.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:22.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:22.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:22.850 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:53:22.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:53:22.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:22.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:22.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:22.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:22.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:22.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:22.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:22.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:22.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:22.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:22.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:23.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:23.320 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:23.321 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:23.321 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:23.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:23.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:23.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:23.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:23.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:23.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:23.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:23.838 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:23.838 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:24.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:24.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:24.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:24.342 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:24.343 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:24.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:24.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:24.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:24.444 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:24.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:24.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:24.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:24.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:24.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:24.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:24.802 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:24.802 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:25.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:25.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:25.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:25.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:25.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:25.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:25.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:25.355 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:25.355 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:25.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:25.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:25.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:25.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:25.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:25.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:25.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:25.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:25.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:25.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:25.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:25.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:25.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:26.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:26.338 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:26.338 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:26.338 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:26.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:26.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:26.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:26.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:26.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:26.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:26.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:26.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:26.865 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:27.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:27.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:27.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:27.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:27.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:27.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:27.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:27.481 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:27.481 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:27.481 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:27.482 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:27.482 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:27.482 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:27.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:27.849 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:27.849 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:27.849 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:28.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:28.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:28.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:28.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:28.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:28.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:28.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:28.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:28.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:28.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:28.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:28.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:28.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:28.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:28.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:28.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:28.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:28.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:28.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:28.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:28.983 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:28.983 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:29.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:29.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:29.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:29.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:29.975 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:29.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:29.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:29.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:29.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:29.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:29.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:29.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:29.978 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:30.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:30.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:30.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:30.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:30.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:30.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:30.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:30.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:30.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:30.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:30.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:30.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:30.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:31.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:31.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:31.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:31.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:31.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:31.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:31.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:31.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:31.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:31.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:31.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:31.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:31.518 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:32.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:32.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:32.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:32.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:32.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:32.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:32.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:32.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:32.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:32.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:32.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:32.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:32.162 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:32.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:32.525 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:32.525 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:32.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:33.123 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:33.123 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:33.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:33.125 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:33.125 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:33.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:53:33.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:53:33.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:33.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:33.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:33.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:33.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:33.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:33.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:33.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:33.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:33.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:33.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:34.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:34.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:34.149 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:34.149 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:34.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:34.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:34.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:34.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:34.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:34.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:34.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:34.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:34.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:35.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:35.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:35.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:35.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:35.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:35.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:35.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:35.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:35.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:35.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:35.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:35.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:35.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:35.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:35.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:35.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:35.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:36.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:36.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:36.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:36.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:36.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:36.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:36.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:36.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:36.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:36.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:36.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:36.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:36.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:36.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:36.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:36.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:36.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:36.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:36.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:36.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:36.861 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:36.861 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:37.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:37.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:37.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:37.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:37.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:37.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:37.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:37.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:37.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:37.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:37.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:37.748 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:37.748 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:38.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:38.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:38.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:38.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:38.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:38.395 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:38.396 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:38.397 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:38.397 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:38.397 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:38.397 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:38.398 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:38.398 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:38.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:38.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:38.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:38.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:39.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:39.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:39.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:39.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:39.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:39.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:39.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:39.267 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:39.267 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:39.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:39.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:39.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:39.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:39.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:39.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:39.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:39.910 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:39.910 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:39.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:39.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:39.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:39.911 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:40.288 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:40.289 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:40.289 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:40.290 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:40.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:40.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:40.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:40.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:40.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:40.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:40.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:40.822 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:40.822 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:41.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:41.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:41.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:41.373 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:41.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:41.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:41.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:41.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:41.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:41.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:41.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:41.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:41.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:41.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:41.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:41.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:42.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:42.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:42.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:42.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:42.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:42.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:42.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:42.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:42.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:42.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:42.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:42.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:42.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:42.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:42.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:42.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:42.961 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:42.961 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:42.962 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:42.962 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:42.962 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:42.962 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:43.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:43.335 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:43.335 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:43.336 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:43.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:43.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:43.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:43.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:43.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:43.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:43.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:43.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:43.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:44.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:44.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:44.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:44.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:44.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:44.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:44.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:44.503 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:44.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:44.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:44.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:44.504 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:44.505 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:44.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:44.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:44.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:44.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:45.428 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:45.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:45.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:45.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:45.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:45.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:45.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:45.430 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:45.431 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:45.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:53:45.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:53:45.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:45.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:45.970 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:46.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:46.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:46.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:46.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:46.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:46.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:46.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:46.072 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:46.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:46.451 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:46.451 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:46.451 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:47.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:47.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:47.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:47.064 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:47.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:47.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:47.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:47.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:47.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:47.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:53:47.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:53:47.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:47.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:47.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:47.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:47.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:47.682 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:47.683 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:47.683 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:47.683 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:47.683 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:47.684 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:48.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:48.052 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:48.052 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:48.053 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:48.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:48.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:48.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:48.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:48.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:48.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:48.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:48.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:48.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:49.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:53:49.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:53:49.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:49.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:49.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:49.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:49.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:49.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:49.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:49.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:49.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:49.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:49.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:49.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:49.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:49.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:49.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:50.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:50.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:50.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:50.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:50.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:50.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:50.647 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:50.647 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:50.647 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:50.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:50.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:50.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:50.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:50.748 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:50.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:50.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:50.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:50.749 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:50.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:51.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:51.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:51.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:51.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:51.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:51.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:51.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:51.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:51.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:51.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:51.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:51.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:51.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:52.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:53:52.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:53:52.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:52.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:52.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:52.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:52.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:52.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:52.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:52.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:52.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:52.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:52.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:52.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:52.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:52.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:52.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:53.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:53.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:53.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:53.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:53.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:53.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:53.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:53.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:53.123 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:53.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:53:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:53:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:53.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:53.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:53.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:53.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:53.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:53.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:53.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:53.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:53.743 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:54.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:54.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:54.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:54.171 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:54.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:54.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:54.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:54.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:54.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:54.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:54.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:54.703 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:54.703 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:55.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:53:55.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:53:55.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:55.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:55.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:55.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:55.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:55.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:55.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:55.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:55.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:55.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:55.371 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:55.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:55.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:55.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:55.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:56.274 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:56.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:56.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:56.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:56.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:56.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:56.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:56.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:56.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:56.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:53:56.821 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:53:56.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:56.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:56.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:56.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:56.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:56.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:56.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:56.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:56.924 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:56.924 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:56.924 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:57.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:57.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:57.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:57.300 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:57.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:57.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:57.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:57.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:57.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:57.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:57.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:57.806 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:57.806 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:58.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:53:58.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:53:58.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:58.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:58.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:58.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:58.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:58.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:58.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:58.446 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:58.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:58.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:58.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:53:58.810 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:53:58.811 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:53:58.811 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:53:58.811 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:53:59.307 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:53:59.308 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:53:59.308 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:53:59.308 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:53:59.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:53:59.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:53:59.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:53:59.310 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:53:59.310 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:53:59.850 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:53:59.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:53:59.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:53:59.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:53:59.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:53:59.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:53:59.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:53:59.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:53:59.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:53:59.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:53:59.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:53:59.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:53:59.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:00.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:00.320 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:00.320 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:00.320 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:00.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:00.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:00.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:00.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:00.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:00.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:00.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:00.826 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:00.826 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:01.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:01.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:01.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:01.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:01.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:01.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:01.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:01.453 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:01.454 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:01.454 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:01.454 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:01.454 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:01.455 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:01.820 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:01.821 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:01.822 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:01.822 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:02.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:02.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:02.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:02.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:02.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:02.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:02.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:02.320 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:02.320 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:02.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:02.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:02.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:02.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:02.852 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:02.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:02.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:02.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:02.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:02.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:02.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:02.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:02.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:03.343 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:03.344 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:03.344 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:03.344 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:03.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:03.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:03.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:03.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:03.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:03.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:03.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:03.861 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:03.861 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:04.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:04.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:04.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:04.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:04.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:04.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:04.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:04.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:04.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:04.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:04.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:04.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:04.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:04.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:04.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:04.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:04.907 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:05.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:05.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:05.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:05.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:05.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:05.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:05.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:05.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:05.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:05.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:05.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:05.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:05.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:05.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:06.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:06.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:06.025 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:06.026 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:06.026 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:06.026 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:06.026 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:06.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:06.392 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:06.393 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:06.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:06.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:06.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:06.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:06.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:06.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:06.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:06.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:06.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:06.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:06.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:07.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:07.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:07.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:07.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:07.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:07.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:07.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:07.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:07.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:07.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:07.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:07.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:07.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:07.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:07.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:07.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:07.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:08.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:08.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:08.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:08.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:08.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:08.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:08.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:08.423 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:08.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:08.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:08.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:08.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:08.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:08.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:09.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:09.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:09.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:09.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:09.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:09.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:09.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:09.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:09.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:09.441 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:09.441 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:09.441 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:09.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:09.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:09.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:09.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:09.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:09.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:09.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:09.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:09.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:10.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:10.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:10.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:10.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:10.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:10.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:10.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:10.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:10.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:10.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:10.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:10.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:10.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:10.994 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:10.994 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:10.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:11.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:11.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:11.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:11.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:11.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:11.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:11.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:11.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:11.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:12.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:54:12.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:54:12.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:12.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:12.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:12.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:12.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:12.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:12.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:12.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:12.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:12.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:12.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:12.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:12.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:12.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:12.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:12.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:12.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:12.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:12.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:12.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:12.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:12.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:12.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:12.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:13.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:54:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:54:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:13.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:13.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:13.613 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:13.613 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:13.613 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:13.614 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:13.614 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:13.614 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:13.979 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:13.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:13.981 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:13.981 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:14.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:14.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:14.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:14.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:14.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:14.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:14.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:14.491 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:14.491 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:15.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:54:15.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:54:15.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:15.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:15.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:15.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:15.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:15.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:15.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:15.486 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:15.486 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:15.486 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:16.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:16.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:16.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:16.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:16.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:16.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:16.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:16.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:16.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:16.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:16.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:16.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:16.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:16.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:16.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:16.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:16.608 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:16.608 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:16.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:16.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:16.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:16.610 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:16.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:16.969 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:16.969 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:16.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:17.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:17.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:17.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:17.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:17.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:17.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:17.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:17.492 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:17.493 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:17.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:17.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:17.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:17.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:17.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:18.098 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:18.099 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:18.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:18.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:18.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:18.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:18.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:18.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:18.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:18.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:18.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:18.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:19.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:19.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:19.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:19.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:19.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:19.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:19.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:19.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:19.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:19.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:54:19.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:54:19.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:19.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:19.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:19.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:19.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:19.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:19.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:19.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:19.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:19.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:19.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:20.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:20.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:20.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:20.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:20.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:20.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:20.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:20.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:20.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:20.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:20.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:20.544 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:20.544 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:21.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:54:21.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:54:21.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:21.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:21.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:21.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:21.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:21.178 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:21.178 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:21.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:21.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:21.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:21.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:21.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:21.546 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:21.546 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:21.547 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:22.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:22.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:22.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:22.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:22.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:22.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:22.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:22.048 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:22.049 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:22.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:22.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:22.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:22.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:22.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:22.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:22.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:22.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:22.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:22.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:22.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:22.672 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:22.672 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:23.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:23.042 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:23.042 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:23.043 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:23.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:23.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:23.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:23.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:23.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:23.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:23.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:23.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:23.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:24.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:24.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:24.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:24.101 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:24.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:24.201 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:24.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:24.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:24.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:24.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:24.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:24.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:24.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:24.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:24.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:24.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:24.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:25.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:25.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:25.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:25.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:25.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:25.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:25.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:25.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:25.085 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:25.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:25.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:25.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:25.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:25.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:25.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:25.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:25.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:25.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:25.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:25.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:25.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:25.764 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:26.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:26.195 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:26.195 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:26.196 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:26.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:26.741 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:26.741 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:26.741 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:26.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:26.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:26.742 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:26.743 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:26.743 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:27.289 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:27.289 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:27.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:27.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:27.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:27.390 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:27.390 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:27.391 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:27.391 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:27.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:27.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:27.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:27.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:27.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:27.784 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:27.784 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:27.785 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:28.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:28.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:28.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:28.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:28.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:28.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:28.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:28.307 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:28.307 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:28.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:28.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:28.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:28.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:28.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:28.945 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:28.946 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:28.947 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:28.947 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:28.947 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:28.947 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:28.948 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:28.948 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:29.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:29.315 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:29.315 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:29.315 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:29.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:29.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:29.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:29.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:29.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:29.810 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:29.810 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:29.810 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:29.811 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:30.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:54:30.307 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:54:30.307 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:30.307 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:30.307 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:30.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:30.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:30.408 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:30.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:30.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:30.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:30.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:30.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:30.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:30.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:30.772 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:30.772 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:31.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:31.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:31.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:31.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:31.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:31.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:31.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:31.356 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:31.356 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:31.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:31.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:31.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:31.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:31.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:31.990 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:31.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:31.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:31.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:31.992 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:31.993 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:31.993 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:31.993 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:32.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:32.355 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:32.355 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:32.356 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:32.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:32.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:32.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:32.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:32.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:32.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:32.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:32.876 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:32.877 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:33.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:33.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:33.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:33.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:33.372 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:33.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:33.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:33.473 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:33.473 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:33.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:33.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:33.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:33.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:33.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:33.835 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:33.835 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:33.836 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:34.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:34.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:34.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:34.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:34.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:34.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:34.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:34.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:34.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:34.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:34.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:34.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:34.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:34.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:34.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:34.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:34.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:34.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:34.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:34.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:34.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:34.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:35.430 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:35.431 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:35.432 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:35.432 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:36.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:36.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:36.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:36.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:36.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:36.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:36.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:36.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:36.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:36.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:36.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:36.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:36.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:36.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:36.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:36.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:36.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:36.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:36.753 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:36.754 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:37.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:37.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:37.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:37.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:37.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:37.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:37.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:37.645 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:37.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:37.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:37.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:37.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:37.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:38.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:38.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:38.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:38.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:38.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:38.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:38.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:38.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:38.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:38.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:38.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:38.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:38.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:38.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:38.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:38.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:38.607 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:39.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:39.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:39.217 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:39.218 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:39.218 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:39.218 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:39.218 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:39.219 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:39.219 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:39.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:39.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:39.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:39.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:39.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:39.870 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:39.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:39.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:39.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:39.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:39.872 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:39.873 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:39.873 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:40.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:40.239 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:40.240 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:40.240 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:40.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:40.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:40.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:40.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:40.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:40.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:40.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:40.760 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:40.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:41.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:41.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:41.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:41.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:41.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:41.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:41.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:41.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:41.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:41.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:41.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:41.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:41.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:41.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:41.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:41.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:41.815 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:42.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:42.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:42.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:42.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:42.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:42.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:42.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:42.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:42.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:42.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:42.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:42.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:42.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:42.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:42.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:42.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:42.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:42.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:42.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:42.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:42.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:42.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:43.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:43.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:43.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:43.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:43.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:43.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:43.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:43.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:43.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:43.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:43.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:43.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:43.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:44.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:44.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:44.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:44.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:44.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:44.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:44.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:44.525 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:44.525 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:44.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:44.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:44.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:44.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:44.888 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:44.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:44.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:44.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:45.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:45.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:45.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:45.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:45.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:45.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:45.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:45.387 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:45.387 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:45.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:45.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:45.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:45.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:45.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:46.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:46.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:46.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:46.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:46.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:46.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:46.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:46.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:46.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:46.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:46.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:46.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:46.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:46.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:46.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:46.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:46.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:46.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:46.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:46.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:46.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:47.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:47.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:47.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:47.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:47.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:47.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:47.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:47.487 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:47.487 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:47.487 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:47.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:47.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:47.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:47.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:47.859 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:47.859 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:47.859 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:48.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:48.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:48.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:48.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:48.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:48.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:48.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:48.365 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:48.365 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:48.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:48.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:48.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:48.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:48.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:49.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:49.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:49.089 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:49.089 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:49.090 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:49.090 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:49.090 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:49.090 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:49.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:49.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:49.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:49.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:49.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:49.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:49.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:49.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:49.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:49.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:49.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:49.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:49.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:50.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:50.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:50.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:50.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:50.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:50.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:50.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:50.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:50.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:50.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:50.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:50.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:50.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:50.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:50.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:50.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:50.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:51.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:51.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:51.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:51.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:51.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:51.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:51.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:51.494 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:51.494 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:52.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:54:52.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:54:52.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:52.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:52.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:52.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:52.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:52.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:52.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:52.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:52.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:52.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:52.141 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:52.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:52.505 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:52.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:52.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:53.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:53.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:53.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:53.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:53.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:53.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:53.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:53.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:53.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:53.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:53.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:53.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:53.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:53.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:53.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:53.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:53.617 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:53.617 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:53.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:53.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:53.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:53.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:53.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:53.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:53.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:53.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:54.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:54.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:54.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:54.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:54.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:54.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:54.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:54.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:54.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:54.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:54.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:54.988 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:54.989 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:54.989 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:55.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:55.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:55.090 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:55.090 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:55.091 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:55.091 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:55.092 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:55.092 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:55.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:55.460 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:55.460 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:55.460 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:55.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:55.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:55.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:55.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:55.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:55.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:55.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:55.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:55.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:56.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:56.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:56.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:56.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:56.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:56.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:56.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:56.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:56.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:56.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:56.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:56.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:56.603 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:57.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:57.005 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:57.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:57.006 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:57.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:57.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:57.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:57.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:57.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:57.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:57.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:57.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:57.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:58.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:54:58.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:54:58.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:58.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:58.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:58.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:58.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:58.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:58.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:58.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:58.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:58.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:58.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:58.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:58.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:58.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:58.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:54:59.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:54:59.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:54:59.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:54:59.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:54:59.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:54:59.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:54:59.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:54:59.009 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:54:59.009 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:54:59.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:54:59.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:54:59.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:54:59.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:54:59.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:54:59.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:54:59.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:54:59.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:54:59.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:54:59.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:54:59.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:54:59.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:54:59.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:54:59.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:54:59.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:54:59.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:54:59.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:00.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:00.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:00.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:00.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:00.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:00.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:00.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:00.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:00.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:00.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:55:00.991 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:55:00.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:00.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:00.992 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:01.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:01.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:01.093 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:01.093 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:01.093 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:01.093 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:01.094 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:01.094 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:01.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:01.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:01.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:01.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:01.970 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:01.970 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:01.970 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:01.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:01.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:01.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:01.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:01.972 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:01.972 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:02.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:02.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:02.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:02.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:02.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:02.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:02.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:02.642 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:02.642 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:02.642 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:02.642 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:02.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:02.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:03.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:03.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:03.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:03.012 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:03.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:03.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:03.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:03.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:03.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:03.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:03.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:03.515 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:03.515 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:04.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:55:04.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:55:04.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:04.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:04.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:04.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:04.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:04.136 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:04.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:04.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:04.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:04.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:04.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:04.510 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:04.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:04.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:04.511 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:05.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:05.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:05.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:05.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:05.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:05.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:05.016 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:05.016 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:05.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:05.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:05.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:05.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:05.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:05.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:05.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:05.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:05.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:05.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:05.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:05.678 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:05.678 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:05.678 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:06.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:06.046 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:06.046 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:06.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:06.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:06.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:06.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:06.550 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:06.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:06.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:06.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:06.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:06.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:07.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:07.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:07.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:07.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:07.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:07.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:07.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:07.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:07.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:07.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:07.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:07.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:07.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:07.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:07.604 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:07.604 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:07.605 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:08.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:08.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:08.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:08.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:08.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:08.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:08.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:08.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:08.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:08.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:08.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:08.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:08.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:08.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:08.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:08.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:08.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:08.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:08.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:08.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:08.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:08.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:09.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:09.168 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:09.168 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:09.168 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:09.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:09.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:09.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:09.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:09.726 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:09.726 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:09.726 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:09.727 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:09.727 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:10.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:55:10.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:55:10.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:10.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:10.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:10.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:10.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:10.329 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:10.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:10.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:10.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:10.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:10.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:10.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:10.722 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:10.722 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:10.723 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:11.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:11.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:11.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:11.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:11.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:11.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:11.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:55:11.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:55:11.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:11.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:11.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:11.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:11.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:11.861 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:11.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:11.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:11.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:11.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:11.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:12.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:12.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:12.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:12.229 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:12.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:12.724 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:12.724 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:12.724 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:12.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:12.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:12.725 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:12.729 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:12.729 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:13.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:55:13.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:55:13.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:13.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:13.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:13.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:13.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:13.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:13.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:13.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:13.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:13.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:13.369 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:13.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:13.740 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:13.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:13.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:14.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:14.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:14.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:14.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:14.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:14.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:14.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:14.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:14.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:14.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:55:14.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:55:14.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:14.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:14.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:14.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:14.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:14.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:14.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:14.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:14.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:14.976 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:14.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:15.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:15.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:15.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:15.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:15.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:15.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:15.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:15.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:15.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:15.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:15.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:15.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:15.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:16.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:16.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:16.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:16.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:16.419 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:16.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:16.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:16.520 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:16.521 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:16.521 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:16.521 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:16.521 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:16.522 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:16.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:16.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:16.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:16.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:17.392 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:17.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:17.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:17.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:17.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:17.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:17.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:17.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:17.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:17.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:17.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:17.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:17.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:17.897 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:17.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:17.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:17.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:17.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:17.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:17.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:17.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:17.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:18.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:18.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:18.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:18.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:18.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:18.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:18.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:18.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:18.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:18.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:18.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:18.962 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:18.962 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:19.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:55:19.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:55:19.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:19.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:19.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:19.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:19.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:19.573 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:19.573 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:19.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:19.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:19.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:19.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:19.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:19.940 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:19.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:19.941 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:20.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:20.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:20.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:20.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:20.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:20.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:20.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:20.460 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:20.461 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:20.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:55:20.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:55:20.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:20.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:20.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:21.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:21.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:21.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:21.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:21.080 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:21.080 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:21.080 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:21.080 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:21.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:21.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:21.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:21.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:21.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:21.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:21.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:21.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:21.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:21.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:21.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:21.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:21.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:22.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:55:22.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:55:22.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:22.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:22.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:22.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:22.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:22.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:22.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:22.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:22.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:22.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:22.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:22.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:22.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:22.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:22.998 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:23.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:23.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:23.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:23.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:23.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:23.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:23.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:23.505 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:23.505 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:24.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:55:24.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:55:24.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:24.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:24.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:24.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:24.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:24.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:24.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:24.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:24.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:24.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:24.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:24.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:24.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:24.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:24.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:25.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:25.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:25.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:25.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:25.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:25.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:25.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:25.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:25.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:25.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:25.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:25.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:25.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:25.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:25.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:25.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:25.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:25.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:25.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:25.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:25.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:25.631 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:25.994 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:25.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:25.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:25.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:26.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:26.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:26.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:26.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:26.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:26.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:26.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:26.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:26.513 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:27.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:27.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:27.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:27.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:27.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:27.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:27.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:27.151 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:27.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:27.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:27.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:27.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:27.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:27.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:27.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:27.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:27.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:28.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:28.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:28.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:28.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:28.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:28.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:28.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:28.009 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:28.009 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:28.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:55:28.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:55:28.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:28.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:28.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:28.607 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:28.608 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:28.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:28.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:28.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:28.610 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:28.610 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:28.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:28.979 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:28.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:28.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:28.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:29.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:29.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:29.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:29.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:29.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:29.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:29.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:29.485 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:29.485 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:30.181 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:55:30.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:55:30.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:30.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:30.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:30.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:30.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:30.285 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:30.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:30.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:30.287 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:30.288 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:30.288 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:30.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:30.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:30.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:30.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:31.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:31.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:31.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:31.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:31.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:31.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:31.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:31.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:31.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:31.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=2293 2021-08-21T14:55:31.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 2293 '!=' null 2021-08-21T14:55:31.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:31.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:31.998 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:32.098 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:32.098 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:32.099 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:32.099 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:32.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:32.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:32.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:32.101 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:32.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:32.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:32.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:32.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:32.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:32.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:32.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:32.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:32.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:32.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:32.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:32.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:32.975 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:33.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=2293 2021-08-21T14:55:33.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 2293 '!=' null 2021-08-21T14:55:33.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:33.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:33.519 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:33.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:33.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:33.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:33.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:33.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:33.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:33.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:33.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:33.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:33.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:33.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:33.984 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:34.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:34.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:34.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:34.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:34.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:34.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:34.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:34.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:34.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:34.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=2293 2021-08-21T14:55:34.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 2293 '!=' null 2021-08-21T14:55:34.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:34.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:34.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:35.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:35.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:35.068 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:35.069 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:35.069 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:35.069 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:35.069 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:35.069 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:35.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:35.429 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:35.429 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:35.430 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:35.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:35.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:35.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:35.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:35.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:35.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:35.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:35.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:35.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:36.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=2293 2021-08-21T14:55:36.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 2293 '!=' null 2021-08-21T14:55:36.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:55:36.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:36.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:55:36.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:36.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:36.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:36.551 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:36.551 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:36.551 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:36.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:36.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:36.907 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:55:36.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:36.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:36.909 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:37.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:55:37.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:55:37.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:55:37.401 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:55:37.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:55:37.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:55:37.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:55:37.403 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:55:37.403 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:55:37.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=null 2021-08-21T14:55:37.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test null '!=' null 2021-08-21T14:55:37.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: (( 1 >= 13 )) 2021-08-21T14:55:37.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:55:37.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.2 2021-08-21T14:55:38.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:55:38.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:55:38.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:55:38.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:55:38.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:55:38.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:55:38.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:55:38.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:55:38.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T14:55:38.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:55:38.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:55:38.470 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:55:38.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 1 2021-08-21T14:55:38.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T14:55:38.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T14:55:38.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2021-08-21T14:55:38.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:55:38.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:55:39.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:55:39.414 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:55:39.414 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:55:39.414 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:55:39.414 INFO:tasks.workunit.client.0.smithi077.stderr:4 2021-08-21T14:55:39.415 INFO:tasks.workunit.client.0.smithi077.stderr:5' 2021-08-21T14:55:39.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:55:39.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:55:39.416 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:55:39.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836589 2021-08-21T14:55:39.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836589 2021-08-21T14:55:39.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836589' 2021-08-21T14:55:39.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:55:39.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:55:39.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673063 2021-08-21T14:55:39.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673063 2021-08-21T14:55:39.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836589 1-42949673063' 2021-08-21T14:55:39.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:55:39.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:55:39.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509541 2021-08-21T14:55:39.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509541 2021-08-21T14:55:39.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509541' 2021-08-21T14:55:39.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:55:39.807 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:55:39.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899346014 2021-08-21T14:55:39.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899346014 2021-08-21T14:55:39.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509541 3-85899346014' 2021-08-21T14:55:39.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:55:39.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:55:40.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182487 2021-08-21T14:55:40.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182487 2021-08-21T14:55:40.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509541 3-85899346014 4-107374182487' 2021-08-21T14:55:40.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:55:40.069 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-08-21T14:55:40.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=128849018957 2021-08-21T14:55:40.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 128849018957 2021-08-21T14:55:40.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836589 1-42949673063 2-64424509541 3-85899346014 4-107374182487 5-128849018957' 2021-08-21T14:55:40.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:55:40.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836589 2021-08-21T14:55:40.200 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:55:40.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:55:40.201 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836589 2021-08-21T14:55:40.201 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:55:40.201 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836589 2021-08-21T14:55:40.202 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836589 2021-08-21T14:55:40.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2021-08-21T14:55:40.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:55:40.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836587 -lt 21474836589 2021-08-21T14:55:40.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:55:41.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:55:41.570 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:55:41.950 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836589 -lt 21474836589 2021-08-21T14:55:41.950 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:55:41.951 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673063 2021-08-21T14:55:41.951 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:55:41.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:55:41.952 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673063 2021-08-21T14:55:41.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:55:41.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673063 2021-08-21T14:55:41.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673063' 2021-08-21T14:55:41.954 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673063 2021-08-21T14:55:41.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:55:42.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673064 -lt 42949673063 2021-08-21T14:55:42.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:55:42.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509541 2021-08-21T14:55:42.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:55:42.369 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:55:42.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509541 2021-08-21T14:55:42.370 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:55:42.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509541 2021-08-21T14:55:42.371 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509541' 2021-08-21T14:55:42.372 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509541 2021-08-21T14:55:42.372 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:55:42.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509541 -lt 64424509541 2021-08-21T14:55:42.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:55:42.813 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899346014 2021-08-21T14:55:42.813 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:55:42.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:55:42.815 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899346014 2021-08-21T14:55:42.815 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:55:42.816 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899346014 2021-08-21T14:55:42.816 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899346014' 2021-08-21T14:55:42.817 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899346014 2021-08-21T14:55:42.817 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:55:43.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899346015 -lt 85899346014 2021-08-21T14:55:43.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:55:43.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182487 2021-08-21T14:55:43.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:55:43.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:55:43.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182487 2021-08-21T14:55:43.206 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:55:43.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182487 2021-08-21T14:55:43.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2021-08-21T14:55:43.208 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182487 2021-08-21T14:55:43.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:55:43.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182488 -lt 107374182487 2021-08-21T14:55:43.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:55:43.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 5-128849018957 2021-08-21T14:55:43.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:55:43.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=5 2021-08-21T14:55:43.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 5-128849018957 2021-08-21T14:55:43.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:55:43.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=128849018957 2021-08-21T14:55:43.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.5 seq 128849018957' 2021-08-21T14:55:43.585 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.5 seq 128849018957 2021-08-21T14:55:43.586 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 5 2021-08-21T14:55:43.989 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 128849018957 -lt 128849018957 2021-08-21T14:55:43.990 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2021-08-21T14:55:43.990 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2021-08-21T14:55:44.365 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:55:44.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2021-08-21T14:55:44.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2021-08-21T14:55:44.375 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:55:44.375 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:55:44.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:55:44.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:55:44.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:55:44.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:55:44.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2021-08-21T14:55:44.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2021-08-21T14:55:44.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:44.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: _objectstore_tool_nodown: shift 2021-08-21T14:55:44.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: _objectstore_tool_nodown: local id=4 2021-08-21T14:55:44.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: _objectstore_tool_nodown: shift 2021-08-21T14:55:44.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2021-08-21T14:55:44.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2021-08-21T14:55:44.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2021-08-21T14:55:46.083 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T14:55:46.083 INFO:tasks.workunit.client.0.smithi077.stdout: "pg_log_t": { 2021-08-21T14:55:46.084 INFO:tasks.workunit.client.0.smithi077.stdout: "head": "34'150", 2021-08-21T14:55:46.084 INFO:tasks.workunit.client.0.smithi077.stdout: "tail": "34'148", 2021-08-21T14:55:46.084 INFO:tasks.workunit.client.0.smithi077.stdout: "log": [ 2021-08-21T14:55:46.084 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.084 INFO:tasks.workunit.client.0.smithi077.stdout: "op": "modify", 2021-08-21T14:55:46.085 INFO:tasks.workunit.client.0.smithi077.stdout: "object": "1:e9c179f9:::obj-149:head", 2021-08-21T14:55:46.085 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'149", 2021-08-21T14:55:46.085 INFO:tasks.workunit.client.0.smithi077.stdout: "prior_version": "0'0", 2021-08-21T14:55:46.086 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4819.0:1", 2021-08-21T14:55:46.086 INFO:tasks.workunit.client.0.smithi077.stdout: "extra_reqids": [], 2021-08-21T14:55:46.086 INFO:tasks.workunit.client.0.smithi077.stdout: "mtime": "2021-08-21T14:50:30.198798+0000", 2021-08-21T14:55:46.086 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": 0, 2021-08-21T14:55:46.087 INFO:tasks.workunit.client.0.smithi077.stdout: "mod_desc": { 2021-08-21T14:55:46.087 INFO:tasks.workunit.client.0.smithi077.stdout: "object_mod_desc": { 2021-08-21T14:55:46.087 INFO:tasks.workunit.client.0.smithi077.stdout: "can_local_rollback": false, 2021-08-21T14:55:46.087 INFO:tasks.workunit.client.0.smithi077.stdout: "rollback_info_completed": false, 2021-08-21T14:55:46.088 INFO:tasks.workunit.client.0.smithi077.stdout: "ops": [] 2021-08-21T14:55:46.088 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.088 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.088 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_regions": { 2021-08-21T14:55:46.088 INFO:tasks.workunit.client.0.smithi077.stdout: "object_clean_regions": { 2021-08-21T14:55:46.089 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_offsets": "[9176~18446744073709542439]", 2021-08-21T14:55:46.089 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_omap": true, 2021-08-21T14:55:46.089 INFO:tasks.workunit.client.0.smithi077.stdout: "new_object": false 2021-08-21T14:55:46.089 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.090 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.090 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.090 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.090 INFO:tasks.workunit.client.0.smithi077.stdout: "op": "modify", 2021-08-21T14:55:46.091 INFO:tasks.workunit.client.0.smithi077.stdout: "object": "1:cb39fd60:::obj-150:head", 2021-08-21T14:55:46.091 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'150", 2021-08-21T14:55:46.091 INFO:tasks.workunit.client.0.smithi077.stdout: "prior_version": "0'0", 2021-08-21T14:55:46.091 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4822.0:1", 2021-08-21T14:55:46.091 INFO:tasks.workunit.client.0.smithi077.stdout: "extra_reqids": [], 2021-08-21T14:55:46.092 INFO:tasks.workunit.client.0.smithi077.stdout: "mtime": "2021-08-21T14:50:30.315402+0000", 2021-08-21T14:55:46.092 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": 0, 2021-08-21T14:55:46.092 INFO:tasks.workunit.client.0.smithi077.stdout: "mod_desc": { 2021-08-21T14:55:46.092 INFO:tasks.workunit.client.0.smithi077.stdout: "object_mod_desc": { 2021-08-21T14:55:46.093 INFO:tasks.workunit.client.0.smithi077.stdout: "can_local_rollback": false, 2021-08-21T14:55:46.093 INFO:tasks.workunit.client.0.smithi077.stdout: "rollback_info_completed": false, 2021-08-21T14:55:46.093 INFO:tasks.workunit.client.0.smithi077.stdout: "ops": [] 2021-08-21T14:55:46.093 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.094 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.094 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_regions": { 2021-08-21T14:55:46.094 INFO:tasks.workunit.client.0.smithi077.stdout: "object_clean_regions": { 2021-08-21T14:55:46.094 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_offsets": "[9176~18446744073709542439]", 2021-08-21T14:55:46.094 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_omap": true, 2021-08-21T14:55:46.095 INFO:tasks.workunit.client.0.smithi077.stdout: "new_object": false 2021-08-21T14:55:46.095 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.095 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.095 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.096 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T14:55:46.096 INFO:tasks.workunit.client.0.smithi077.stdout: "dups": [ 2021-08-21T14:55:46.096 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.096 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4795.0:1", 2021-08-21T14:55:46.097 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'141", 2021-08-21T14:55:46.097 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "141", 2021-08-21T14:55:46.097 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.097 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.097 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.098 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4798.0:1", 2021-08-21T14:55:46.098 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'142", 2021-08-21T14:55:46.098 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "142", 2021-08-21T14:55:46.098 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.099 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.099 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.099 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4801.0:1", 2021-08-21T14:55:46.099 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'143", 2021-08-21T14:55:46.100 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "143", 2021-08-21T14:55:46.100 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.100 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.100 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.100 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4804.0:1", 2021-08-21T14:55:46.101 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'144", 2021-08-21T14:55:46.101 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "144", 2021-08-21T14:55:46.101 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.102 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.102 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.102 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4807.0:1", 2021-08-21T14:55:46.102 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'145", 2021-08-21T14:55:46.103 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "145", 2021-08-21T14:55:46.103 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.103 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.103 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.104 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4810.0:1", 2021-08-21T14:55:46.104 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'146", 2021-08-21T14:55:46.104 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "146", 2021-08-21T14:55:46.104 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.104 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.105 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.105 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4813.0:1", 2021-08-21T14:55:46.105 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'147", 2021-08-21T14:55:46.106 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "147", 2021-08-21T14:55:46.106 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.106 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.106 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T14:55:46.107 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4816.0:1", 2021-08-21T14:55:46.107 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'148", 2021-08-21T14:55:46.107 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "148", 2021-08-21T14:55:46.107 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T14:55:46.107 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.108 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T14:55:46.108 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T14:55:46.108 INFO:tasks.workunit.client.0.smithi077.stdout: "pg_missing_t": { 2021-08-21T14:55:46.108 INFO:tasks.workunit.client.0.smithi077.stdout: "missing": [], 2021-08-21T14:55:46.109 INFO:tasks.workunit.client.0.smithi077.stdout: "may_include_deletes": true 2021-08-21T14:55:46.109 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T14:55:46.109 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T14:55:46.110 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2021-08-21T14:55:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2021-08-21T14:55:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2021-08-21T14:55:46.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2021-08-21T14:55:46.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=8 2021-08-21T14:55:46.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: '[' 8 '!=' 8 ']' 2021-08-21T14:55:46.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2021-08-21T14:55:46.113 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to max 2 2021-08-21T14:55:46.113 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to copy log version 34'150 2021-08-21T14:55:46.113 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to copy log version 34'149 2021-08-21T14:55:46.113 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after earliest_dup_version 141 2021-08-21T14:55:46.114 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T14:55:46.114 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T14:55:46.114 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T14:55:46.115 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T14:55:46.115 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T14:55:46.115 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T14:55:46.115 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T14:55:46.116 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T14:55:46.116 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to max 2 2021-08-21T14:55:46.116 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to copy log version 34'150 2021-08-21T14:55:46.116 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to copy log version 34'149 2021-08-21T14:55:46.117 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after earliest_dup_version 141 2021-08-21T14:55:46.117 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T14:55:46.117 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T14:55:46.117 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T14:55:46.118 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T14:55:46.118 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T14:55:46.118 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T14:55:46.118 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T14:55:46.118 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T14:55:46.119 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to max 2 2021-08-21T14:55:46.119 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to copy log version 34'150 2021-08-21T14:55:46.119 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to copy log version 34'149 2021-08-21T14:55:46.120 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after earliest_dup_version 141 2021-08-21T14:55:46.120 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T14:55:46.120 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T14:55:46.121 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T14:55:46.126 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T14:55:46.126 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T14:55:46.127 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T14:55:46.127 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T14:55:46.127 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:50:33.264+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T14:55:46.128 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_after v 34'150 2021-08-21T14:55:46.129 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after earliest_dup_version 141 2021-08-21T14:55:46.129 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T14:55:46.129 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T14:55:46.129 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T14:55:46.130 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T14:55:46.130 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T14:55:46.130 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T14:55:46.130 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T14:55:46.131 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T14:55:46.131 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'149 2021-08-21T14:55:46.131 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:55:30.221+0000 7fe914ecb700 20 copy_up_to/copy_after copy dup from log version 34'150 2021-08-21T14:55:46.132 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2021-08-21T14:55:46.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2021-08-21T14:55:46.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2021-08-21T14:55:46.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:55:46.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-08-21T14:55:46.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:55:46.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:55:46.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:55:46.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:55:46.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:55:46.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:55:46.136 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:55:46.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:55:46.136 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:55:46.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:55:46.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:55:46.138 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:55:46.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:55:46.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:55:46.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:55:46.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:55:46.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:55:46.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:55:46.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-08-21T14:55:46.191 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:55:46.191 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:46.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:46.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53763 2021-08-21T14:55:46.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:55:46.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:55:46.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-08-21T14:55:46.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-08-21T14:55:46.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-08-21T14:55:46.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.196 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T14:55:46.196 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-08-21T14:55:46.196 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T14:55:46.196 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T14:55:46.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T14:55:46.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T14:55:46.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T14:55:46.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T14:55:46.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T14:55:46.199 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T14:55:46.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T14:55:46.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T14:55:46.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T14:55:46.200 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T14:55:46.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:55:46.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T14:55:46.201 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T14:55:46.201 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T14:55:46.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T14:55:46.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T14:55:46.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-08-21T14:55:46.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T14:55:46.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:46.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:46.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53763 2021-08-21T14:55:46.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T14:55:46.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T14:55:46.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-08-21T14:55:46.207 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-08-21T14:55:46.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:46.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:46.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53763 2021-08-21T14:55:46.208 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-08-21T14:55:46.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-08-21T14:55:46.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-08-21T14:55:46.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-08-21T14:55:46.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2021-08-21T14:55:46.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:115: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2021-08-21T14:55:46.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:55:46.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2021-08-21T14:55:46.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2021-08-21T14:55:46.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2021-08-21T14:55:46.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2021-08-21T14:55:46.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-08-21T14:55:46.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-08-21T14:55:46.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-08-21T14:55:46.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-08-21T14:55:46.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-08-21T14:55:46.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-08-21T14:55:46.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-08-21T14:55:46.583 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-08-21T14:55:46.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:55:46.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:55:46.585 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:55:46.585 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:46.586 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:46.586 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:55:46.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-08-21T14:55:46.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-08-21T14:55:46.828 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-08-21T14:55:46.829 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:55:46.829 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:55:46.829 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=fsid 2021-08-21T14:55:46.829 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .fsid 2021-08-21T14:55:46.830 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:55:46.830 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:55:46.830 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:55:46.830 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:55:46.831 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:46.831 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:46.831 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53763/ceph-mon.a.asok 2021-08-21T14:55:46.831 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:55:46.831 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.53763/ceph-mon.a.asok config get fsid 2021-08-21T14:55:46.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-08-21T14:55:46.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T14:55:46.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T14:55:46.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=mon_host 2021-08-21T14:55:46.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .mon_host 2021-08-21T14:55:46.930 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T14:55:46.930 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T14:55:46.931 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T14:55:46.931 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T14:55:46.931 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:46.931 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:46.932 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53763/ceph-mon.a.asok 2021-08-21T14:55:46.932 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T14:55:46.932 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.53763/ceph-mon.a.asok config get mon_host 2021-08-21T14:55:47.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-08-21T14:55:47.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:47.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-08-21T14:55:47.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-08-21T14:55:47.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-08-21T14:55:47.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-08-21T14:55:47.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-08-21T14:55:47.897 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-08-21T14:55:47.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:55:47.898 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:55:47.898 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:55:47.899 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:47.899 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:47.899 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:55:47.899 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:55:47.900 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T14:55:47.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-08-21T14:55:47.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:55:47.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:55:47.924 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-08-21T14:55:47.925 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-08-21T14:55:47.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:55:47.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-08-21T14:55:47.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:55:47.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:55:47.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-08-21T14:55:47.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:55:47.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-08-21T14:55:47.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:55:47.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:55:47.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:55:47.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:55:47.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-08-21T14:55:47.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-08-21T14:55:47.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:55:47.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:55:47.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:55:47.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:55:47.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:55:47.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:55:47.933 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:55:47.934 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:55:47.934 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:55:47.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:55:47.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:55:47.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:55:47.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:55:47.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:55:47.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:55:47.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:55:47.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:55:47.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:55:47.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:55:47.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:55:47.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:55:47.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-08-21T14:55:47.940 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:55:47.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=1cbc9676-e948-4dbf-bb03-a302d0b1bc9d 2021-08-21T14:55:47.941 INFO:tasks.workunit.client.0.smithi077.stdout:add osd0 1cbc9676-e948-4dbf-bb03-a302d0b1bc9d 2021-08-21T14:55:47.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 1cbc9676-e948-4dbf-bb03-a302d0b1bc9d' 2021-08-21T14:55:47.943 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:55:47.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQDzEyFhlmo4OBAAGK/e8ME4hyf/9HH3lQp3WA== 2021-08-21T14:55:47.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQDzEyFhlmo4OBAAGK/e8ME4hyf/9HH3lQp3WA=="}' 2021-08-21T14:55:47.945 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 1cbc9676-e948-4dbf-bb03-a302d0b1bc9d -i td/osd-backfill-recovery-log/0/new.json 2021-08-21T14:55:48.327 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:55:48.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-08-21T14:55:48.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDzEyFhlmo4OBAAGK/e8ME4hyf/9HH3lQp3WA== --osd-uuid 1cbc9676-e948-4dbf-bb03-a302d0b1bc9d 2021-08-21T14:55:48.364 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.362+0000 7f2bb3fde0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:55:48.365 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.364+0000 7f2bb3fde0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:55:48.366 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.365+0000 7f2bb3fde0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:55:48.367 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.366+0000 7f2bb3fde0c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T14:55:48.367 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.366+0000 7f2bb3fde0c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T14:55:48.367 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.366+0000 7f2bb3fde0c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T14:55:48.418 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:48.417+0000 7f2bb3fde0c0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-08-21T14:55:54.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-08-21T14:55:54.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:55:54.158 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd0 key to auth repository 2021-08-21T14:55:54.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-08-21T14:55:54.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:55:54.667 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-08-21T14:55:54.668 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.0 2021-08-21T14:55:54.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:55:54.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:55:54.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:55:54.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:55:54.691 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:54.690+0000 7fbe700010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:55:54.715 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:54.713+0000 7fbe700010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:55:54.715 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:54.715+0000 7fbe700010c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:55:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-08-21T14:55:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:55:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=0 2021-08-21T14:55:55.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:55:55.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:55:55.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:55:55.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:55:55.062 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:55:55.062 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:55:55.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:55:55.237 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:55.236+0000 7fbe700010c0 -1 Falling back to public interface 2021-08-21T14:55:55.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:55:56.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:55:56.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:55:56.497 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:55:56.498 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:55:56.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:55:56.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:55:56.946 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:55:57.660 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:55:57.659+0000 7fbe700010c0 -1 osd.0 0 log_to_monitors {default=true} 2021-08-21T14:55:57.947 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:55:57.947 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:55:57.948 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:55:57.948 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:55:57.949 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:55:57.949 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:55:58.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:55:59.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:55:59.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:55:59.351 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:55:59.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:55:59.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:55:59.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:55:59.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:00.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:00.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:00.774 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:56:00.774 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:56:00.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:00.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:01.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:02.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:02.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:02.204 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:56:02.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:56:02.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:02.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:02.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:03.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:03.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:03.637 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:56:03.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:56:03.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:03.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:04.021 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:05.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:05.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:05.023 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:56:05.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:56:05.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:05.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:05.426 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:06.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:06.427 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:06.428 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:56:06.428 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:56:06.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:06.429 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:06.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:07.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:07.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:07.835 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:56:07.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:56:07.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:07.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:08.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:09.348 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:09.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:09.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:56:09.349 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:56:09.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:09.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:09.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:10.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:10.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:10.825 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:56:10.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:56:10.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:10.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:11.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:12.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:12.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:12.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:56:12.352 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:56:12.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:12.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:12.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:13.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:13.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:13.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:56:13.882 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:56:13.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:13.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:14.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:15.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:15.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:15.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:56:15.352 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:56:15.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:15.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:15.851 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:16.853 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:16.853 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:16.853 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:56:16.854 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:56:16.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:16.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:17.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:18.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:18.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:18.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:56:18.664 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:56:18.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:18.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:19.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:20.204 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:56:20.205 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:56:20.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:20.206 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T14:56:20.625 INFO:tasks.workunit.client.0.smithi077.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85766,v1:127.0.0.1:6803/85766] [v2:127.0.0.1:6804/85766,v1:127.0.0.1:6805/85766] exists,up 1cbc9676-e948-4dbf-bb03-a302d0b1bc9d 2021-08-21T14:56:20.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:56:20.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:56:20.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:56:20.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:56:20.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-08-21T14:56:20.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:56:20.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:56:20.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-08-21T14:56:20.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:56:20.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-08-21T14:56:20.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:56:20.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:56:20.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:56:20.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:56:20.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-08-21T14:56:20.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-08-21T14:56:20.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:56:20.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:56:20.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:56:20.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:56:20.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:56:20.636 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:56:20.636 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:56:20.636 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:56:20.636 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:56:20.636 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:56:20.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:56:20.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:56:20.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:56:20.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:56:20.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:56:20.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:56:20.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:56:20.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:56:20.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:56:20.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:56:20.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:56:20.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-08-21T14:56:20.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:56:20.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=bf35496b-5cde-4bfb-971e-4f071cbb0c35 2021-08-21T14:56:20.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 bf35496b-5cde-4bfb-971e-4f071cbb0c35' 2021-08-21T14:56:20.641 INFO:tasks.workunit.client.0.smithi077.stdout:add osd1 bf35496b-5cde-4bfb-971e-4f071cbb0c35 2021-08-21T14:56:20.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:56:20.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAUFCFhwPRXJhAA5lUjgH2vpDmhYWY0Q1mDDA== 2021-08-21T14:56:20.644 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAUFCFhwPRXJhAA5lUjgH2vpDmhYWY0Q1mDDA=="}' 2021-08-21T14:56:20.644 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new bf35496b-5cde-4bfb-971e-4f071cbb0c35 -i td/osd-backfill-recovery-log/1/new.json 2021-08-21T14:56:21.099 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:56:21.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-08-21T14:56:21.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAUFCFhwPRXJhAA5lUjgH2vpDmhYWY0Q1mDDA== --osd-uuid bf35496b-5cde-4bfb-971e-4f071cbb0c35 2021-08-21T14:56:21.136 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.134+0000 7f2a6543c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:21.159 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.157+0000 7f2a6543c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:21.159 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.158+0000 7f2a6543c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:21.160 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.158+0000 7f2a6543c0c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T14:56:21.160 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.158+0000 7f2a6543c0c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T14:56:21.160 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.158+0000 7f2a6543c0c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T14:56:21.198 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:21.197+0000 7f2a6543c0c0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-08-21T14:56:26.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-08-21T14:56:26.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:56:26.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-08-21T14:56:26.873 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd1 key to auth repository 2021-08-21T14:56:26.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:56:27.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-08-21T14:56:27.415 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.1 2021-08-21T14:56:27.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:56:27.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:56:27.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:56:27.417 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:56:27.437 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:27.436+0000 7faaca73d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:27.453 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:27.452+0000 7faaca73d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:27.454 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:27.453+0000 7faaca73d0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:27.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-08-21T14:56:27.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:56:27.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=1 2021-08-21T14:56:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:56:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:56:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:27.828 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:56:27.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:56:27.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:27.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:27.981 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:27.979+0000 7faaca73d0c0 -1 Falling back to public interface 2021-08-21T14:56:28.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:29.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:29.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:29.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:56:29.270 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:56:29.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:29.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:29.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:30.357 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:30.356+0000 7faaca73d0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-08-21T14:56:30.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:30.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:30.654 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:56:30.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:56:30.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:30.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:31.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:32.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:32.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:32.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:56:32.179 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:56:32.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:32.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:32.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:33.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:33.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:33.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:56:33.561 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:56:33.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:33.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:33.941 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:34.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:34.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:34.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:56:34.943 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:56:34.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:34.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:35.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:36.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:36.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:36.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:56:36.437 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:56:36.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:36.438 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:36.856 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:37.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:37.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:37.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:56:37.858 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:56:37.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:37.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:38.255 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:39.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:39.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:39.258 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:56:39.258 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:56:39.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:39.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:39.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:40.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:40.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:40.751 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:56:40.752 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:56:40.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:40.752 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:41.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:42.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:42.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:42.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:56:42.278 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:56:42.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:42.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:43.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:44.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:44.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:44.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:56:44.091 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:56:44.094 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:44.094 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:45.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:46.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:46.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:46.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:56:46.160 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:56:46.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:46.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:46.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:47.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:47.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:47.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:56:47.859 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:56:47.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:47.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:48.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:49.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:49.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:49.303 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:56:49.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:56:49.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:49.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:49.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:50.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:50.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:50.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:56:50.793 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:56:50.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:50.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:51.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:56:52.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:56:52.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:56:52.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:56:52.847 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:56:52.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:56:52.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T14:56:53.231 INFO:tasks.workunit.client.0.smithi077.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/87590,v1:127.0.0.1:6811/87590] [v2:127.0.0.1:6812/87590,v1:127.0.0.1:6813/87590] exists,up bf35496b-5cde-4bfb-971e-4f071cbb0c35 2021-08-21T14:56:53.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:56:53.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:56:53.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:56:53.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:56:53.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-08-21T14:56:53.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:56:53.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:56:53.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-08-21T14:56:53.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:56:53.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-08-21T14:56:53.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:56:53.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:56:53.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:56:53.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:56:53.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-08-21T14:56:53.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-08-21T14:56:53.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:56:53.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:56:53.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:56:53.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:56:53.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:56:53.241 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:56:53.241 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:56:53.242 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:56:53.242 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:56:53.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:56:53.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:56:53.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:56:53.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:56:53.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:56:53.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:56:53.243 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:56:53.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:56:53.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:56:53.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:56:53.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:56:53.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:56:53.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-08-21T14:56:53.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:56:53.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=a603f427-6a1f-4fee-b4c2-9716bcb87103 2021-08-21T14:56:53.246 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 a603f427-6a1f-4fee-b4c2-9716bcb87103' 2021-08-21T14:56:53.249 INFO:tasks.workunit.client.0.smithi077.stdout:add osd2 a603f427-6a1f-4fee-b4c2-9716bcb87103 2021-08-21T14:56:53.250 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:56:53.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQA1FCFhvn3LDhAAQuYcxdLxzjmbcr8Z/O7u7w== 2021-08-21T14:56:53.251 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQA1FCFhvn3LDhAAQuYcxdLxzjmbcr8Z/O7u7w=="}' 2021-08-21T14:56:53.251 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new a603f427-6a1f-4fee-b4c2-9716bcb87103 -i td/osd-backfill-recovery-log/2/new.json 2021-08-21T14:56:53.864 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:56:53.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-08-21T14:56:53.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA1FCFhvn3LDhAAQuYcxdLxzjmbcr8Z/O7u7w== --osd-uuid a603f427-6a1f-4fee-b4c2-9716bcb87103 2021-08-21T14:56:53.901 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.900+0000 7f44dc9f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:53.922 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.921+0000 7f44dc9f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:53.924 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.922+0000 7f44dc9f80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:56:53.924 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.922+0000 7f44dc9f80c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T14:56:53.925 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.922+0000 7f44dc9f80c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T14:56:53.925 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.922+0000 7f44dc9f80c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T14:56:53.999 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:56:53.999+0000 7f44dc9f80c0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-08-21T14:57:00.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-08-21T14:57:00.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:57:00.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-08-21T14:57:00.116 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd2 key to auth repository 2021-08-21T14:57:00.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:57:00.615 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.2 2021-08-21T14:57:00.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-08-21T14:57:00.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:57:00.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:57:00.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:57:00.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:57:00.634 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:00.634+0000 7f865e7d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:00.635 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:00.635+0000 7f865e7d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:00.636 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:00.636+0000 7f865e7d90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:01.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-08-21T14:57:01.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:57:01.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=2 2021-08-21T14:57:01.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:57:01.053 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:57:01.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:01.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:57:01.055 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:57:01.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:01.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:01.154 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:01.153+0000 7f865e7d90c0 -1 Falling back to public interface 2021-08-21T14:57:01.411 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:02.411 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:02.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:02.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:57:02.412 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:57:02.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:02.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:02.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:03.620 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:03.619+0000 7f865e7d90c0 -1 osd.2 0 log_to_monitors {default=true} 2021-08-21T14:57:03.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:03.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:03.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:57:03.841 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:57:03.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:03.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:04.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:05.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:05.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:05.319 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:57:05.319 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:57:05.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:05.320 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:05.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:06.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:06.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:06.777 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:57:06.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:57:06.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:06.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:07.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:08.258 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:08.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:08.260 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:57:08.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:57:08.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:08.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:08.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:09.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:09.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:09.737 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:57:09.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:57:09.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:09.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:10.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:11.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:11.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:11.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:57:11.138 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:57:11.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:11.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:11.674 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:12.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:12.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:12.676 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:57:12.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:57:12.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:12.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:13.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:14.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:14.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:14.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:57:14.082 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:57:14.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:14.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:14.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:15.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:15.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:15.540 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:57:15.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:57:15.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:15.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:16.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:17.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:17.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:17.364 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:57:17.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:57:17.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:17.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:17.972 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:18.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:18.976 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:18.977 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:57:18.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:57:18.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:18.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:19.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:20.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:20.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:20.877 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:57:20.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:57:20.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:20.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:21.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:22.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:22.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:22.473 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:57:22.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:57:22.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:22.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:23.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:24.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:24.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:24.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:57:24.142 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:57:24.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:24.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:25.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:26.025 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:26.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:26.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:57:26.026 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:57:26.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:26.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T14:57:26.438 INFO:tasks.workunit.client.0.smithi077.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/89367,v1:127.0.0.1:6819/89367] [v2:127.0.0.1:6820/89367,v1:127.0.0.1:6821/89367] exists,up a603f427-6a1f-4fee-b4c2-9716bcb87103 2021-08-21T14:57:26.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:57:26.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:57:26.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:57:26.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:57:26.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2021-08-21T14:57:26.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:57:26.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:57:26.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2021-08-21T14:57:26.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:57:26.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2021-08-21T14:57:26.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:57:26.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:57:26.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:57:26.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:57:26.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2021-08-21T14:57:26.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2021-08-21T14:57:26.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:57:26.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:57:26.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:57:26.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:57:26.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:57:26.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:57:26.448 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:57:26.448 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:57:26.449 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:57:26.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:57:26.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:57:26.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:57:26.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:57:26.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:57:26.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:57:26.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:57:26.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:57:26.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:57:26.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:57:26.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:57:26.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:57:26.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2021-08-21T14:57:26.452 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:57:26.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=f4916d08-b1a7-4757-a411-9117c5930247 2021-08-21T14:57:26.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd3 f4916d08-b1a7-4757-a411-9117c5930247' 2021-08-21T14:57:26.453 INFO:tasks.workunit.client.0.smithi077.stdout:add osd3 f4916d08-b1a7-4757-a411-9117c5930247 2021-08-21T14:57:26.453 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:57:26.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBWFCFhrPQoGxAAaVCfNGyrdDF9zmEBxK9I1A== 2021-08-21T14:57:26.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBWFCFhrPQoGxAAaVCfNGyrdDF9zmEBxK9I1A=="}' 2021-08-21T14:57:26.457 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new f4916d08-b1a7-4757-a411-9117c5930247 -i td/osd-backfill-recovery-log/3/new.json 2021-08-21T14:57:27.096 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:57:27.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2021-08-21T14:57:27.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 3 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBWFCFhrPQoGxAAaVCfNGyrdDF9zmEBxK9I1A== --osd-uuid f4916d08-b1a7-4757-a411-9117c5930247 2021-08-21T14:57:27.135 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.132+0000 7fc2f80250c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:27.155 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.154+0000 7fc2f80250c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:27.156 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.155+0000 7fc2f80250c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:27.157 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.155+0000 7fc2f80250c0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-08-21T14:57:27.157 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.155+0000 7fc2f80250c0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-08-21T14:57:27.157 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.155+0000 7fc2f80250c0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2021-08-21T14:57:27.231 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:27.230+0000 7fc2f80250c0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2021-08-21T14:57:32.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2021-08-21T14:57:32.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:57:32.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd3 key to auth repository 2021-08-21T14:57:32.963 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd3 key to auth repository 2021-08-21T14:57:32.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:57:33.467 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.3 2021-08-21T14:57:33.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.3 2021-08-21T14:57:33.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 3 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:57:33.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:57:33.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:57:33.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:57:33.491 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:33.489+0000 7fe84b55f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:33.492 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:33.491+0000 7fe84b55f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:33.493 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:33.492+0000 7fe84b55f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:57:33.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 3 2021-08-21T14:57:33.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:57:33.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=3 2021-08-21T14:57:33.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:57:33.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:57:33.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:33.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:57:33.923 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:57:33.923 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:33.924 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:34.025 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:34.024+0000 7fe84b55f0c0 -1 Falling back to public interface 2021-08-21T14:57:34.308 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:35.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:35.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:35.311 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:57:35.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:57:35.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:35.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:35.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:36.412 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:57:36.411+0000 7fe84b55f0c0 -1 osd.3 0 log_to_monitors {default=true} 2021-08-21T14:57:36.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:36.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:36.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:57:36.696 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:57:36.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:36.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:37.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:38.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:38.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:38.238 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:57:38.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:57:38.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:38.239 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:38.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:39.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:39.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:39.635 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:57:39.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:57:39.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:39.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:40.066 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:41.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:41.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:41.068 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:57:41.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:57:41.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:41.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:41.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:42.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:42.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:42.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:57:42.519 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:57:42.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:42.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:42.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:43.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:43.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:43.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:57:43.902 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:57:43.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:43.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:44.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:45.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:45.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:45.352 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:57:45.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:57:45.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:45.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:45.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:46.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:46.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:46.737 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:57:46.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:57:46.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:46.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:47.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:48.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:48.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:48.227 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:57:48.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:57:48.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:48.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:49.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:50.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:50.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:50.005 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:57:50.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:57:50.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:50.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:51.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:52.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:52.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:52.127 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:57:52.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:57:52.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:52.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:52.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:53.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:53.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:53.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:57:53.698 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:57:53.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:53.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:54.205 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:55.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:55.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:55.207 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:57:55.208 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:57:55.208 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:55.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:55.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:56.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:56.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:56.715 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:57:56.715 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:57:56.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:56.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:57.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:57:58.702 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:57:58.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:57:58.703 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:57:58.703 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:57:58.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:57:58.704 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:57:59.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:00.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:00.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:00.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:58:00.144 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:58:00.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:00.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.3 up' 2021-08-21T14:58:00.569 INFO:tasks.workunit.client.0.smithi077.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/91116,v1:127.0.0.1:6827/91116] [v2:127.0.0.1:6828/91116,v1:127.0.0.1:6829/91116] exists,up f4916d08-b1a7-4757-a411-9117c5930247 2021-08-21T14:58:00.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:58:00.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:58:00.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:58:00.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:58:00.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2021-08-21T14:58:00.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:58:00.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:58:00.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2021-08-21T14:58:00.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:58:00.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2021-08-21T14:58:00.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:58:00.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:58:00.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:58:00.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:58:00.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2021-08-21T14:58:00.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2021-08-21T14:58:00.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:58:00.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:58:00.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:58:00.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:58:00.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:58:00.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:58:00.585 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:58:00.585 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:58:00.585 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:58:00.585 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:58:00.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:58:00.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:58:00.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:58:00.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:58:00.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:58:00.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:58:00.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:58:00.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:58:00.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:58:00.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:58:00.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:58:00.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2021-08-21T14:58:00.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:58:00.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=712cc994-0264-46ba-b29a-63f9cf2e96aa 2021-08-21T14:58:00.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd4 712cc994-0264-46ba-b29a-63f9cf2e96aa' 2021-08-21T14:58:00.589 INFO:tasks.workunit.client.0.smithi077.stdout:add osd4 712cc994-0264-46ba-b29a-63f9cf2e96aa 2021-08-21T14:58:00.590 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:58:00.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQB4FCFhhfrwIhAATDa+ouzlcEY7vsViilztWw== 2021-08-21T14:58:00.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQB4FCFhhfrwIhAATDa+ouzlcEY7vsViilztWw=="}' 2021-08-21T14:58:00.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 712cc994-0264-46ba-b29a-63f9cf2e96aa -i td/osd-backfill-recovery-log/4/new.json 2021-08-21T14:58:01.076 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:58:01.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2021-08-21T14:58:01.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 4 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB4FCFhhfrwIhAATDa+ouzlcEY7vsViilztWw== --osd-uuid 712cc994-0264-46ba-b29a-63f9cf2e96aa 2021-08-21T14:58:01.115 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.113+0000 7f3a36af80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:01.116 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.115+0000 7f3a36af80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:01.117 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.115+0000 7f3a36af80c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:01.117 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.116+0000 7f3a36af80c0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-08-21T14:58:01.118 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.116+0000 7f3a36af80c0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-08-21T14:58:01.118 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.116+0000 7f3a36af80c0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2021-08-21T14:58:01.191 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:01.191+0000 7f3a36af80c0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2021-08-21T14:58:07.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2021-08-21T14:58:07.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:58:07.181 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd4 key to auth repository 2021-08-21T14:58:07.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd4 key to auth repository 2021-08-21T14:58:07.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:58:07.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.4 2021-08-21T14:58:07.686 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.4 2021-08-21T14:58:07.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 4 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:58:07.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:58:07.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:58:07.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:58:07.708 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:07.707+0000 7fba19f3c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:07.710 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:07.709+0000 7fba19f3c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:07.711 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:07.710+0000 7fba19f3c0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:08.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 4 2021-08-21T14:58:08.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:58:08.126 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=4 2021-08-21T14:58:08.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:58:08.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:58:08.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:08.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:58:08.128 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:58:08.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:08.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:08.232 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:08.230+0000 7fba19f3c0c0 -1 Falling back to public interface 2021-08-21T14:58:08.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:09.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:09.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:09.537 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:58:09.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:58:09.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:09.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:09.919 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:10.659 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:10.658+0000 7fba19f3c0c0 -1 osd.4 0 log_to_monitors {default=true} 2021-08-21T14:58:10.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:10.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:10.921 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:58:10.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:58:10.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:10.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:11.328 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:12.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:12.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:12.330 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:58:12.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:58:12.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:12.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:12.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:13.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:13.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:13.877 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:58:13.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:58:13.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:13.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:14.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:15.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:15.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:15.274 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:58:15.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:58:15.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:15.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:15.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:16.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:16.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:16.680 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:58:16.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:58:16.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:16.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:17.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:18.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:18.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:18.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:58:18.088 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:58:18.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:18.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:18.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:19.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:19.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:19.612 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:58:19.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:58:19.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:19.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:20.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:21.030 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:21.030 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:21.030 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:58:21.032 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:58:21.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:21.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:21.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:22.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:22.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:22.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:58:22.665 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:58:22.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:22.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:23.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:24.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:24.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:24.229 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:58:24.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:58:24.230 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:24.230 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:24.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:25.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:25.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:25.691 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:58:25.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:58:25.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:25.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:26.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:27.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:27.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:27.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:58:27.356 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:58:27.356 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:27.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:27.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:28.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:28.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:28.876 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:58:28.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:58:28.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:28.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:29.782 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:30.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:30.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:30.784 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:58:30.784 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:58:30.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:30.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:31.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:32.501 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:32.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:32.502 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:58:32.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:58:32.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:32.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:33.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:34.131 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:34.132 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:34.132 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:58:34.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:58:34.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:34.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.4 up' 2021-08-21T14:58:34.509 INFO:tasks.workunit.client.0.smithi077.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/92899,v1:127.0.0.1:6835/92899] [v2:127.0.0.1:6836/92899,v1:127.0.0.1:6837/92899] exists,up 712cc994-0264-46ba-b29a-63f9cf2e96aa 2021-08-21T14:58:34.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:58:34.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:58:34.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:58:34.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T14:58:34.532 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2021-08-21T14:58:34.532 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T14:58:34.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T14:58:34.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2021-08-21T14:58:34.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T14:58:34.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2021-08-21T14:58:34.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T14:58:34.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T14:58:34.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T14:58:34.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T14:58:34.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2021-08-21T14:58:34.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2021-08-21T14:58:34.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T14:58:34.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2021-08-21T14:58:34.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T14:58:34.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T14:58:34.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T14:58:34.540 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T14:58:34.541 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T14:58:34.541 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T14:58:34.541 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T14:58:34.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:58:34.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T14:58:34.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T14:58:34.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T14:58:34.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T14:58:34.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T14:58:34.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T14:58:34.546 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T14:58:34.546 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T14:58:34.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:58:34.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T14:58:34.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T14:58:34.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2021-08-21T14:58:34.548 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T14:58:34.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=5f8f3a93-5e17-4636-9f2a-7d7698b283c7 2021-08-21T14:58:34.549 INFO:tasks.workunit.client.0.smithi077.stdout:add osd5 5f8f3a93-5e17-4636-9f2a-7d7698b283c7 2021-08-21T14:58:34.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd5 5f8f3a93-5e17-4636-9f2a-7d7698b283c7' 2021-08-21T14:58:34.552 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T14:58:34.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQCaFCFh3R5bHxAAuN1hgPXXQpM/Wh/GXwriIA== 2021-08-21T14:58:34.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQCaFCFh3R5bHxAAuN1hgPXXQpM/Wh/GXwriIA=="}' 2021-08-21T14:58:34.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 5f8f3a93-5e17-4636-9f2a-7d7698b283c7 -i td/osd-backfill-recovery-log/5/new.json 2021-08-21T14:58:34.995 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:58:35.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2021-08-21T14:58:35.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 5 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCaFCFh3R5bHxAAuN1hgPXXQpM/Wh/GXwriIA== --osd-uuid 5f8f3a93-5e17-4636-9f2a-7d7698b283c7 2021-08-21T14:58:35.031 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.031+0000 7feb249b70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:35.034 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.034+0000 7feb249b70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:35.034 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.035+0000 7feb249b70c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:35.035 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.035+0000 7feb249b70c0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-08-21T14:58:35.035 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.035+0000 7feb249b70c0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-08-21T14:58:35.036 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.035+0000 7feb249b70c0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2021-08-21T14:58:35.128 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:35.118+0000 7feb249b70c0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2021-08-21T14:58:40.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2021-08-21T14:58:40.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T14:58:40.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd5 key to auth repository 2021-08-21T14:58:40.872 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd5 key to auth repository 2021-08-21T14:58:40.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T14:58:41.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.5 2021-08-21T14:58:41.365 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.5 2021-08-21T14:58:41.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 5 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T14:58:41.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T14:58:41.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T14:58:41.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T14:58:41.393 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:41.387+0000 7fd2bda6b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:41.393 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:41.389+0000 7fd2bda6b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:41.403 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:41.390+0000 7fd2bda6b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T14:58:41.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 5 2021-08-21T14:58:41.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T14:58:41.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=5 2021-08-21T14:58:41.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T14:58:41.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T14:58:41.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:41.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T14:58:41.782 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T14:58:41.782 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:41.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:41.911 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:41.909+0000 7fd2bda6b0c0 -1 Falling back to public interface 2021-08-21T14:58:42.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:43.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:43.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:43.147 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T14:58:43.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T14:58:43.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:43.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:43.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:44.326 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T14:58:44.325+0000 7fd2bda6b0c0 -1 osd.5 0 log_to_monitors {default=true} 2021-08-21T14:58:44.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:44.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:44.573 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T14:58:44.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T14:58:44.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:44.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:45.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:46.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:46.063 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:46.064 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T14:58:46.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T14:58:46.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:46.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:46.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:47.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:47.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:47.537 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T14:58:47.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T14:58:47.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:47.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:47.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:48.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:48.934 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:48.934 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T14:58:48.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T14:58:48.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:48.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:49.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:50.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:50.327 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:50.328 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T14:58:50.329 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T14:58:50.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:50.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:50.710 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:51.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:51.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:51.713 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T14:58:51.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T14:58:51.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:51.714 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:52.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:53.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:53.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:53.230 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T14:58:53.231 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T14:58:53.232 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:53.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:53.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:54.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:54.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:54.655 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T14:58:54.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T14:58:54.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:54.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:55.036 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:56.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:56.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:56.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T14:58:56.038 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T14:58:56.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:56.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:56.770 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:57.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:57.771 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:57.772 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T14:58:57.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T14:58:57.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:57.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:58.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:58:59.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:58:59.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:58:59.292 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T14:58:59.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T14:58:59.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:58:59.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:58:59.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:59:00.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:59:00.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:59:00.930 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T14:59:00.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T14:59:00.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:59:00.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:59:01.908 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:59:02.908 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:59:02.909 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:59:02.910 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T14:59:02.911 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T14:59:02.911 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:59:02.911 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:59:03.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:59:04.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:59:04.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:59:04.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T14:59:04.604 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T14:59:04.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:59:04.606 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:59:05.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:59:06.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:59:06.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:59:06.524 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T14:59:06.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T14:59:06.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:59:06.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:59:07.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T14:59:08.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T14:59:08.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T14:59:08.004 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T14:59:08.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T14:59:08.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T14:59:08.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.5 up' 2021-08-21T14:59:08.383 INFO:tasks.workunit.client.0.smithi077.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/94674,v1:127.0.0.1:6843/94674] [v2:127.0.0.1:6844/94674,v1:127.0.0.1:6845/94674] exists,up 5f8f3a93-5e17-4636-9f2a-7d7698b283c7 2021-08-21T14:59:08.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T14:59:08.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T14:59:08.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T14:59:08.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2021-08-21T14:59:08.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2021-08-21T14:59:09.211 INFO:tasks.workunit.client.0.smithi077.stderr:pool 'test' already exists 2021-08-21T14:59:09.221 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2021-08-21T14:59:10.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2021-08-21T14:59:10.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:10.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2021-08-21T14:59:10.392 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:10.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2021-08-21T14:59:10.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:10.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2021-08-21T14:59:10.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:10.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2021-08-21T14:59:10.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:10.870 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2021-08-21T14:59:10.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:10.987 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2021-08-21T14:59:11.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2021-08-21T14:59:11.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.281 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2021-08-21T14:59:11.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2021-08-21T14:59:11.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2021-08-21T14:59:11.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2021-08-21T14:59:11.709 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.710 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2021-08-21T14:59:11.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2021-08-21T14:59:11.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:11.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2021-08-21T14:59:12.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2021-08-21T14:59:12.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2021-08-21T14:59:12.287 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.287 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2021-08-21T14:59:12.412 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2021-08-21T14:59:12.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2021-08-21T14:59:12.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2021-08-21T14:59:12.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2021-08-21T14:59:12.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:12.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2021-08-21T14:59:13.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2021-08-21T14:59:13.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2021-08-21T14:59:13.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2021-08-21T14:59:13.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2021-08-21T14:59:13.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2021-08-21T14:59:13.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.622 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2021-08-21T14:59:13.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2021-08-21T14:59:13.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2021-08-21T14:59:13.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:13.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2021-08-21T14:59:14.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2021-08-21T14:59:14.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2021-08-21T14:59:14.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2021-08-21T14:59:14.402 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2021-08-21T14:59:14.511 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.512 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2021-08-21T14:59:14.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2021-08-21T14:59:14.754 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2021-08-21T14:59:14.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:14.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2021-08-21T14:59:15.127 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2021-08-21T14:59:15.269 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2021-08-21T14:59:15.370 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.370 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2021-08-21T14:59:15.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2021-08-21T14:59:15.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.599 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2021-08-21T14:59:15.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2021-08-21T14:59:15.805 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.806 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2021-08-21T14:59:15.939 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:15.940 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2021-08-21T14:59:16.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2021-08-21T14:59:16.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2021-08-21T14:59:16.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2021-08-21T14:59:16.367 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.368 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2021-08-21T14:59:16.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2021-08-21T14:59:16.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2021-08-21T14:59:16.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2021-08-21T14:59:16.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2021-08-21T14:59:16.946 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:16.946 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2021-08-21T14:59:17.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2021-08-21T14:59:17.189 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.190 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2021-08-21T14:59:17.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2021-08-21T14:59:17.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2021-08-21T14:59:17.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2021-08-21T14:59:17.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2021-08-21T14:59:17.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2021-08-21T14:59:17.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:17.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2021-08-21T14:59:18.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2021-08-21T14:59:18.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2021-08-21T14:59:18.254 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.254 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2021-08-21T14:59:18.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.355 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2021-08-21T14:59:18.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2021-08-21T14:59:18.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2021-08-21T14:59:18.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2021-08-21T14:59:18.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2021-08-21T14:59:18.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:18.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2021-08-21T14:59:19.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2021-08-21T14:59:19.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2021-08-21T14:59:19.311 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2021-08-21T14:59:19.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2021-08-21T14:59:19.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2021-08-21T14:59:19.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2021-08-21T14:59:19.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2021-08-21T14:59:19.920 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:19.921 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2021-08-21T14:59:20.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2021-08-21T14:59:20.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2021-08-21T14:59:20.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2021-08-21T14:59:20.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2021-08-21T14:59:20.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2021-08-21T14:59:20.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2021-08-21T14:59:20.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.728 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2021-08-21T14:59:20.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.846 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2021-08-21T14:59:20.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:20.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2021-08-21T14:59:21.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2021-08-21T14:59:21.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2021-08-21T14:59:21.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2021-08-21T14:59:21.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.491 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2021-08-21T14:59:21.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2021-08-21T14:59:21.710 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.710 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2021-08-21T14:59:21.818 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.819 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2021-08-21T14:59:21.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:21.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2021-08-21T14:59:22.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2021-08-21T14:59:22.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2021-08-21T14:59:22.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2021-08-21T14:59:22.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2021-08-21T14:59:22.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2021-08-21T14:59:22.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2021-08-21T14:59:22.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:22.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2021-08-21T14:59:23.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2021-08-21T14:59:23.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.268 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2021-08-21T14:59:23.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2021-08-21T14:59:23.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2021-08-21T14:59:23.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2021-08-21T14:59:23.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2021-08-21T14:59:23.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2021-08-21T14:59:23.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:23.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2021-08-21T14:59:24.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2021-08-21T14:59:24.181 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2021-08-21T14:59:24.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2021-08-21T14:59:24.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.442 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2021-08-21T14:59:24.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.551 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2021-08-21T14:59:24.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2021-08-21T14:59:24.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2021-08-21T14:59:24.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:24.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2021-08-21T14:59:25.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2021-08-21T14:59:25.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.238 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2021-08-21T14:59:25.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2021-08-21T14:59:25.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2021-08-21T14:59:25.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2021-08-21T14:59:25.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2021-08-21T14:59:25.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.783 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2021-08-21T14:59:25.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:25.926 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2021-08-21T14:59:26.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2021-08-21T14:59:26.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2021-08-21T14:59:26.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2021-08-21T14:59:26.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2021-08-21T14:59:26.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2021-08-21T14:59:26.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2021-08-21T14:59:26.739 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.739 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2021-08-21T14:59:26.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.848 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2021-08-21T14:59:26.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:26.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2021-08-21T14:59:27.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2021-08-21T14:59:27.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2021-08-21T14:59:27.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2021-08-21T14:59:27.460 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2021-08-21T14:59:27.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2021-08-21T14:59:27.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2021-08-21T14:59:27.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2021-08-21T14:59:27.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:27.905 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2021-08-21T14:59:28.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:28.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2021-08-21T14:59:28.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:28.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2021-08-21T14:59:28.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:28.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2021-08-21T14:59:28.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2021-08-21T14:59:28.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2021-08-21T14:59:28.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2021-08-21T14:59:28.551 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2021-08-21T14:59:28.900 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T14:59:28.910 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2021-08-21T14:59:29.455 INFO:tasks.workunit.client.0.smithi077.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2021-08-21T14:59:29.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2021-08-21T14:59:29.464 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2021-08-21T14:59:30.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2021-08-21T14:59:30.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: wait_for_clean: local cmd= 2021-08-21T14:59:30.466 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: wait_for_clean: local num_active_clean=-1 2021-08-21T14:59:30.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: wait_for_clean: local cur_active_clean 2021-08-21T14:59:30.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2021-08-21T14:59:30.467 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: get_timeout_delays 90 .1 2021-08-21T14:59:30.467 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: shopt -q -o xtrace 2021-08-21T14:59:30.467 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: echo true 2021-08-21T14:59:30.468 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: get_timeout_delays: local trace=true 2021-08-21T14:59:30.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: true 2021-08-21T14:59:30.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: get_timeout_delays: shopt -u -o xtrace 2021-08-21T14:59:30.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: wait_for_clean: local -a delays 2021-08-21T14:59:30.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: wait_for_clean: local -i loop=0 2021-08-21T14:59:30.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: wait_for_clean: flush_pg_stats 2021-08-21T14:59:30.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T14:59:30.570 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T14:59:31.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T14:59:31.039 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T14:59:31.040 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T14:59:31.040 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T14:59:31.040 INFO:tasks.workunit.client.0.smithi077.stderr:4 2021-08-21T14:59:31.040 INFO:tasks.workunit.client.0.smithi077.stderr:5' 2021-08-21T14:59:31.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T14:59:31.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:59:31.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T14:59:31.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836530 2021-08-21T14:59:31.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836530 2021-08-21T14:59:31.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836530' 2021-08-21T14:59:31.229 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:59:31.229 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T14:59:31.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673003 2021-08-21T14:59:31.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673003 2021-08-21T14:59:31.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836530 1-42949673003' 2021-08-21T14:59:31.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:59:31.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T14:59:31.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509477 2021-08-21T14:59:31.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509477 2021-08-21T14:59:31.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836530 1-42949673003 2-64424509477' 2021-08-21T14:59:31.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:59:31.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T14:59:31.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899345946 2021-08-21T14:59:31.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899345946 2021-08-21T14:59:31.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836530 1-42949673003 2-64424509477 3-85899345946' 2021-08-21T14:59:31.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:59:31.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T14:59:31.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182429 2021-08-21T14:59:31.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182429 2021-08-21T14:59:31.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836530 1-42949673003 2-64424509477 3-85899345946 4-107374182429' 2021-08-21T14:59:31.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T14:59:31.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-08-21T14:59:32.014 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=128849018896 2021-08-21T14:59:32.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 128849018896 2021-08-21T14:59:32.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836530 1-42949673003 2-64424509477 3-85899345946 4-107374182429 5-128849018896' 2021-08-21T14:59:32.015 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:59:32.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836530 2021-08-21T14:59:32.016 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:59:32.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T14:59:32.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836530 2021-08-21T14:59:32.017 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:59:32.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836530 2021-08-21T14:59:32.018 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2021-08-21T14:59:32.018 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836530 2021-08-21T14:59:32.018 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:59:32.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836528 -lt 21474836530 2021-08-21T14:59:32.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T14:59:33.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T14:59:33.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T14:59:33.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836530 -lt 21474836530 2021-08-21T14:59:33.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:59:33.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673003 2021-08-21T14:59:33.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:59:33.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T14:59:33.789 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673003 2021-08-21T14:59:33.789 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:59:33.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673003 2021-08-21T14:59:33.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673003' 2021-08-21T14:59:33.790 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673003 2021-08-21T14:59:33.791 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T14:59:34.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673003 -lt 42949673003 2021-08-21T14:59:34.226 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:59:34.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509477 2021-08-21T14:59:34.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:59:34.227 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T14:59:34.231 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509477 2021-08-21T14:59:34.232 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:59:34.232 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509477 2021-08-21T14:59:34.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2021-08-21T14:59:34.233 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509477 2021-08-21T14:59:34.233 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T14:59:34.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509477 -lt 64424509477 2021-08-21T14:59:34.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:59:34.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899345946 2021-08-21T14:59:34.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:59:34.644 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T14:59:34.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899345946 2021-08-21T14:59:34.645 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:59:34.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899345946 2021-08-21T14:59:34.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899345946' 2021-08-21T14:59:34.647 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899345946 2021-08-21T14:59:34.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T14:59:35.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899345946 -lt 85899345946 2021-08-21T14:59:35.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:59:35.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182429 2021-08-21T14:59:35.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:59:35.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T14:59:35.076 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182429 2021-08-21T14:59:35.076 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:59:35.076 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182429 2021-08-21T14:59:35.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182429' 2021-08-21T14:59:35.077 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182429 2021-08-21T14:59:35.078 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T14:59:35.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182430 -lt 107374182429 2021-08-21T14:59:35.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T14:59:35.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 5-128849018896 2021-08-21T14:59:35.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T14:59:35.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=5 2021-08-21T14:59:35.450 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 5-128849018896 2021-08-21T14:59:35.450 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T14:59:35.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=128849018896 2021-08-21T14:59:35.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.5 seq 128849018896' 2021-08-21T14:59:35.451 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.5 seq 128849018896 2021-08-21T14:59:35.452 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 5 2021-08-21T14:59:35.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 128849018897 -lt 128849018896 2021-08-21T14:59:35.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: get_num_pgs 2021-08-21T14:59:35.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:35.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:36.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: wait_for_clean: test 1 == 0 2021-08-21T14:59:36.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:36.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:36.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:36.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:36.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:36.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:36.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:36.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:36.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:36.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:36.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:37.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:37.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' -1 2021-08-21T14:59:37.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1638: wait_for_clean: loop=0 2021-08-21T14:59:37.280 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: num_active_clean=0 2021-08-21T14:59:37.281 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:37.281 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:37.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:37.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:37.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:37.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:37.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:37.383 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:37.383 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:37.383 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:37.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:37.799 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:37.799 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:37.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:38.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:38.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:38.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:38.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:38.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:38.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:38.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:38.317 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:38.318 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:38.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=1109 2021-08-21T14:59:38.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 1109 '!=' null 2021-08-21T14:59:38.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:38.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:38.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:38.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:38.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:38.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:38.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:38.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:38.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:38.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:38.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:39.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:39.301 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:39.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:39.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:39.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:39.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:39.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:39.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:39.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:39.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:39.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:39.796 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:39.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:40.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=1109 2021-08-21T14:59:40.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 1109 '!=' null 2021-08-21T14:59:40.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:40.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:40.332 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:40.432 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:40.432 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:40.433 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:40.433 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:40.433 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:40.433 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:40.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:40.434 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:40.798 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:40.798 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:40.798 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:40.799 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:41.300 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:41.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:41.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:41.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:41.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:41.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:41.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:41.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:41.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:41.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3669 2021-08-21T14:59:41.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3669 '!=' null 2021-08-21T14:59:41.836 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:41.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:41.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:41.937 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:41.938 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:41.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:41.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:41.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:41.939 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:41.939 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:41.939 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:42.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:42.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:42.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:42.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:42.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:42.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:42.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:42.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:42.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:42.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:42.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:42.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:42.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:43.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3106 2021-08-21T14:59:43.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3106 '!=' null 2021-08-21T14:59:43.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:43.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:43.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:43.475 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:43.476 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:43.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:43.477 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:43.477 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:43.477 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:43.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:43.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:43.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:43.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:43.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:43.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:44.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:44.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:44.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:44.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:44.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:44.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:44.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:44.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:44.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:44.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:59:44.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:59:44.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:44.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:44.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:44.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:44.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:44.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:44.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:44.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:44.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:44.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:44.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:45.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:45.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:45.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:45.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:45.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:45.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:45.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:45.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:45.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:45.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:45.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:45.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:45.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:46.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T14:59:46.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T14:59:46.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:46.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:46.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:46.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:46.515 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:46.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:46.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:46.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:46.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:46.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:46.518 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:46.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:46.886 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:46.886 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:46.886 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:47.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:47.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:47.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:47.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:47.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:47.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:47.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:47.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:47.380 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:47.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T14:59:47.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T14:59:47.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:47.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:47.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:47.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:47.978 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:47.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:47.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:47.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:47.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:47.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:47.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:48.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:48.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:48.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:48.347 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:48.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:48.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:48.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:48.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:48.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:48.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:48.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:48.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:48.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:49.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:59:49.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:59:49.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:49.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:49.441 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:49.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:49.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:49.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:49.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:49.542 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:49.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:49.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:49.543 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:49.908 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:49.909 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:49.909 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:49.909 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:50.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:50.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:50.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:50.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:50.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:50.408 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:50.409 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:50.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:50.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:50.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:59:50.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:59:50.942 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:50.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:50.943 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:51.044 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:51.044 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:51.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:51.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:51.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:51.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:51.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:51.045 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:51.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:51.404 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:51.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:51.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:51.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:51.899 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:51.900 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:51.900 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:51.900 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:51.900 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:51.901 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:51.901 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:51.901 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:52.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T14:59:52.393 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T14:59:52.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:52.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:52.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:52.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:52.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:52.495 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:52.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:52.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:52.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:52.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:52.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:52.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:52.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:52.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:52.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:53.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:53.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:53.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:53.357 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:53.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:53.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:53.358 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:53.358 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:53.359 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:54.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:59:54.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:59:54.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:54.087 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:54.088 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:54.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:54.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:54.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:54.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:54.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:54.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:54.190 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:54.190 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:54.562 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:54.562 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:54.563 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:54.563 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:55.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:55.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:55.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:55.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:55.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:55.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T14:59:55.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T14:59:55.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:55.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:55.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:55.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:55.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:55.743 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:55.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:55.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:55.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:55.744 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:55.745 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:56.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:56.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:56.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:56.119 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:56.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:56.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:56.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:56.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:56.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:56.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:56.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:56.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:56.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:57.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:59:57.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:59:57.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:57.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:57.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:57.220 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:57.221 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:57.221 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:57.221 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:57.221 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:57.221 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:57.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:57.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:57.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:57.585 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:57.586 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:57.586 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:58.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:58.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:58.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:58.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:58.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:58.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:58.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:58.092 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:58.093 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T14:59:58.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T14:59:58.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T14:59:58.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T14:59:58.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T14:59:58.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T14:59:58.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T14:59:58.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T14:59:58.732 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T14:59:58.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T14:59:58.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T14:59:58.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T14:59:58.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T14:59:58.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T14:59:59.100 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T14:59:59.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T14:59:59.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T14:59:59.100 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T14:59:59.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T14:59:59.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T14:59:59.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T14:59:59.610 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T14:59:59.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T14:59:59.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T14:59:59.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T14:59:59.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T14:59:59.611 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:00.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:00.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:00.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:00.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:00.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:00.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:00.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:00.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:00.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:00.203 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:00.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:00.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:00.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:00.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:00.566 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:00.566 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:00.566 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:01.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:01.077 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:01.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:01.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:01.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:01.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:01.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:01.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:01.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:01.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:01.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:01.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:01.574 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:01.575 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:01.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:01.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:01.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:01.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:01.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:01.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:01.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:01.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:02.036 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:02.036 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:02.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:02.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:02.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:02.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:02.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:02.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:02.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:02.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:02.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:02.526 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:02.527 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:03.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:03.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:03.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:03.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:03.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:03.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:03.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:03.159 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:03.159 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:03.159 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:03.159 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:03.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:03.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:03.546 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:03.546 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:03.547 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:03.547 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:04.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:04.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:04.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:04.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:04.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:04.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:04.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:04.052 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:04.053 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:04.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:04.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:04.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:04.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:04.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:04.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:04.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:04.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:04.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:04.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:04.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:04.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:04.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:05.061 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:05.062 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:05.062 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:05.063 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:05.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:05.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:05.561 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:05.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:05.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:06.066 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:06.066 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:06.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:06.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:06.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:06.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:06.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:06.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:06.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:06.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:06.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:06.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:06.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:06.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:06.548 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:06.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:06.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:07.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:07.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:07.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:07.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:07.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:07.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:07.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:07.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:07.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:07.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:07.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:07.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:07.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:07.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:07.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:07.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:07.628 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:07.628 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:07.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:07.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:07.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:07.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:07.993 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:07.993 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:07.993 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:07.994 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:08.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:08.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:08.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:08.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:08.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:08.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:08.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:08.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:08.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:09.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:00:09.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:00:09.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:09.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:09.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:09.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:09.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:09.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:09.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:09.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:09.131 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:09.132 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:09.132 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:09.502 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:09.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:09.502 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:09.503 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:10.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:10.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:10.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:10.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:10.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:10.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:10.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:10.007 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:10.008 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:10.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:00:10.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:00:10.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:10.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:10.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:10.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:10.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:10.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:10.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:10.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:10.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:10.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:10.734 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:11.216 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:11.216 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:11.217 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:11.217 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:11.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:11.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:11.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:11.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:11.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:11.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:11.724 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:11.724 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:11.724 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:12.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:12.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:12.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:12.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:12.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:12.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:12.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:12.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:12.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:12.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:12.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:12.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:12.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:12.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:12.751 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:12.751 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:12.751 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:13.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:13.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:13.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:13.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:13.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:13.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:13.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:13.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:13.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:13.802 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:13.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:13.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:13.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:13.804 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:13.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:13.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:13.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:13.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:13.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:13.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:13.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:13.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:14.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:14.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:14.277 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:14.278 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:14.784 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:14.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:14.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:14.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:14.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:14.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:14.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:14.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:14.787 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:15.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:15.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:15.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:15.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:15.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:15.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:15.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:15.405 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:15.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:15.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:15.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:15.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:15.407 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:15.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:15.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:15.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:15.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:16.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:16.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:16.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:16.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:16.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:16.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:16.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:16.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:16.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:16.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:16.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:16.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:16.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:16.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:16.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:16.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:16.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:16.893 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:16.893 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:16.893 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:16.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:16.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:17.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:17.260 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:17.260 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:17.260 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:17.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:17.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:17.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:17.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:17.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:17.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:17.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:17.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:17.789 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:18.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:18.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:18.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:18.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:18.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:18.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:18.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:18.387 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:18.387 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:18.387 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:18.387 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:18.388 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:18.388 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:18.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:18.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:18.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:18.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:19.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:19.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:19.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:19.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:19.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:19.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:19.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:19.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:19.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:19.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:19.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:19.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:19.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:19.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:19.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:19.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:19.933 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:19.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:19.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:19.934 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:19.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:19.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:20.301 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:20.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:20.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:20.302 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:20.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:20.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:20.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:20.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:20.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:20.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:20.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:20.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:20.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:21.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:21.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:21.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:21.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:21.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:21.439 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:21.440 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:21.440 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:21.440 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:21.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:21.442 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:21.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:21.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:21.810 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:21.810 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:21.817 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:21.817 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:22.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:22.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:22.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:22.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:22.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:22.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:22.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:22.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:22.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:22.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:00:22.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:00:22.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:22.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:22.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:22.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:22.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:22.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:22.995 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:22.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:22.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:22.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:22.996 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:23.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:23.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:23.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:23.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:23.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:23.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:23.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:23.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:23.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:23.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:23.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:23.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:23.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:24.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:00:24.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:00:24.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:24.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:24.451 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:24.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:24.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:24.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:24.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:24.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:24.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:24.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:24.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:24.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:24.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:24.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:24.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:25.452 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:25.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:25.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:25.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:25.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:25.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:25.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:25.454 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:25.455 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:25.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:00:25.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:00:25.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:25.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:25.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:26.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:26.065 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:26.065 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:26.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:26.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:26.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:26.066 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:26.067 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:26.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:26.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:26.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:26.445 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:26.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:26.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:26.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:26.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:26.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:26.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:26.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:26.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:26.954 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:27.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:27.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:27.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:27.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:27.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:27.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:27.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:27.590 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:27.590 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:27.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:27.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:27.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:27.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:27.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:27.955 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:27.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:27.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:28.467 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:28.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:28.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:28.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:28.468 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:28.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:28.469 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:28.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:28.469 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:29.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:29.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:29.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:29.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:29.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:29.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:29.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:29.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:29.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:29.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:29.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:29.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:29.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:29.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:29.473 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:29.474 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:29.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:29.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:29.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:29.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:29.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:29.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:29.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:29.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:29.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:29.982 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:30.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:30.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:30.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:30.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:30.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:30.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:30.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:30.587 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:30.587 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:30.587 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:30.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:30.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:30.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:30.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:30.983 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:30.983 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:30.983 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:31.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:31.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:31.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:31.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:31.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:31.568 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:31.568 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:32.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:32.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:32.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:32.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:32.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:32.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:32.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:32.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:32.242 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:32.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:32.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:32.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:32.243 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:32.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:32.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:32.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:32.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:33.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:33.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:33.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:33.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:33.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:33.126 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:33.126 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:33.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:33.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:33.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:33.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:33.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:33.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:33.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:33.762 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:33.762 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:33.762 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:33.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:33.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:33.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:34.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:34.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:34.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:34.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:34.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:34.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:34.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:34.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:34.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:34.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:34.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:34.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:34.630 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:35.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:35.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:35.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:35.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:35.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:35.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:35.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:35.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:35.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:35.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:35.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:35.238 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:35.238 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:35.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:35.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:35.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:35.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:36.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:36.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:36.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:36.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:36.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:36.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:36.122 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:36.123 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:36.123 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:36.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:36.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:36.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:36.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:36.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:36.739 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:36.739 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:36.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:36.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:36.740 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:36.740 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:36.740 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:36.740 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:37.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:37.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:37.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:37.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:37.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:37.616 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:37.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:37.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:37.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:37.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:37.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:37.618 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:37.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:38.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:38.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:38.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:38.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:38.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:38.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:38.271 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:38.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:38.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:38.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:38.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:38.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:38.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:38.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:38.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:38.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:38.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:39.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:39.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:39.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:39.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:39.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:39.136 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:39.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:39.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:39.137 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:39.706 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:00:39.706 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:00:39.707 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:39.707 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:39.708 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:39.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:39.808 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:39.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:39.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:39.808 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:39.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:39.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:39.809 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:40.190 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:40.191 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:40.191 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:40.191 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:40.693 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:40.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:40.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:40.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:40.694 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:40.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:40.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:40.695 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:40.695 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:41.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:41.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:41.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:41.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:41.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:41.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:41.391 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:41.391 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:41.391 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:41.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:41.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:41.392 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:41.393 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:41.753 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:41.754 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:41.754 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:41.754 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:42.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:42.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:42.250 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:42.251 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:42.251 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:42.251 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:42.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:42.252 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:42.252 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:42.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:00:42.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:00:42.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:42.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:42.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:42.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:42.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:42.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:42.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:42.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:42.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:42.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:42.884 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:43.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:43.275 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:43.275 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:43.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:43.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:43.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:43.786 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:43.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:43.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:43.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:43.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:43.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:43.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:44.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:44.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:44.317 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:44.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:44.318 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:44.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:44.418 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:44.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:44.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:44.419 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:44.419 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:44.419 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:44.419 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:44.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:44.791 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:44.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:44.792 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:45.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:45.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:45.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:45.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:45.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:45.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:45.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:45.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:45.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:45.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:45.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:45.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:45.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:45.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:45.979 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:45.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:45.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:45.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:45.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:45.981 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:45.981 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:45.981 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:46.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:46.345 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:46.345 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:46.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:46.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:46.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:46.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:46.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:46.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:46.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:46.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:46.843 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:46.844 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:47.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:00:47.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:00:47.346 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:47.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:47.347 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:47.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:47.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:47.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:47.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:47.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:47.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:47.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:47.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:47.816 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:47.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:47.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:47.817 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:48.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:48.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:48.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:48.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:48.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:48.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:48.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:48.323 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:48.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:48.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:00:48.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:00:48.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:48.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:48.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:48.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:48.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:48.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:48.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:48.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:48.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:48.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:48.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:49.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:49.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:49.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:49.307 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:49.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:49.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:49.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:49.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:49.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:49.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:49.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:49.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:49.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:50.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:00:50.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:00:50.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:50.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:50.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:50.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:50.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:50.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:50.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:50.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:50.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:50.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:50.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:50.904 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:50.904 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:50.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:50.905 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:51.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:51.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:51.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:51.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:51.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:51.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:51.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:51.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:51.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:51.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:00:51.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:00:51.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:51.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:51.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:52.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:52.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:52.033 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:52.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:52.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:52.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:52.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:52.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:52.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:52.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:52.406 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:52.407 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:52.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:52.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:52.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:52.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:52.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:52.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:52.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:52.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:52.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:53.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:53.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:53.438 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:53.438 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:53.438 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:53.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:53.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:53.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:53.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:53.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:53.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:53.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:53.539 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:53.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:53.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:53.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:53.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:54.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:54.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:54.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:54.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:54.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:54.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:54.450 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:54.451 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:54.451 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:54.981 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:54.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:54.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:54.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:54.982 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:55.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:55.083 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:55.083 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:55.083 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:55.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:55.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:55.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:55.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:55.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:55.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:55.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:55.443 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:55.949 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:55.949 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:55.950 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:55.950 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:55.950 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:55.950 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:55.951 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:55.951 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:55.951 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:56.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:56.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:56.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:56.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:56.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:56.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:56.549 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:56.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:56.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:56.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:56.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:56.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:56.550 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:56.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:56.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:56.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:56.923 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:57.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:57.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:57.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:57.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:57.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:57.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:57.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:57.422 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:57.422 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:57.973 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:00:57.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:00:57.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:57.974 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:57.975 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:58.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:58.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:58.075 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:58.076 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:58.076 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:58.076 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:58.076 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:58.077 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:58.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:58.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:58.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:58.450 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:00:58.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:00:58.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:00:58.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:00:58.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:00:58.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:00:58.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:00:58.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:00:58.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:00:58.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:00:59.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:00:59.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:00:59.458 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:00:59.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:00:59.459 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:00:59.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:00:59.560 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:00:59.560 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:00:59.560 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:00:59.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:00:59.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:00:59.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:00:59.561 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:00:59.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:00:59.930 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:00:59.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:00:59.931 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:00.445 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:00.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:00.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:00.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:00.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:00.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:00.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:00.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:00.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:00.952 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:00.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:00.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:00.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:00.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:01.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:01.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:01.054 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:01.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:01.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:01.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:01.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:01.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:01.461 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:01.462 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:01.462 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:01.463 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:01.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:01.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:01.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:01.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:01.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:01.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:01.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:01.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:01.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:02.485 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:02.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:02.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:02.486 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:02.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:02.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:02.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:02.587 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:02.587 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:02.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:02.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:02.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:02.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:02.977 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:02.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:02.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:02.977 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:03.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:03.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:03.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:03.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:03.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:03.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:03.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:03.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:03.501 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:04.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:04.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:04.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:04.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:04.039 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:04.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:04.140 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:04.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:04.140 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:04.141 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:04.141 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:04.141 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:04.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:04.508 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:04.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:04.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:04.509 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:05.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:05.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:05.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:05.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:05.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:05.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:05.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:05.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:05.029 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:05.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:05.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:05.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:05.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:05.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:05.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:05.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:05.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:05.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:05.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:05.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:05.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:05.640 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:06.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:06.011 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:06.011 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:06.011 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:06.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:06.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:06.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:06.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:06.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:06.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:06.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:06.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:06.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:07.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:01:07.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:01:07.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:07.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:07.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:07.180 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:07.181 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:07.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:07.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:07.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:07.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:07.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:07.182 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:07.564 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:07.564 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:07.565 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:07.565 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:08.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:08.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:08.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:08.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:08.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:08.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:08.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:08.070 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:08.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:08.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:01:08.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:01:08.572 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:08.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:08.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:08.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:08.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:08.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:08.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:08.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:08.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:08.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:08.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:09.036 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:09.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:09.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:09.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:09.582 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:09.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:09.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:09.583 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:09.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:09.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:09.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:09.585 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:09.585 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:10.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:10.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:10.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:10.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:10.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:10.210 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:10.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:10.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:10.211 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:10.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:10.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:10.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:10.212 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:10.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:10.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:10.578 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:10.578 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:11.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:11.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:11.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:11.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:11.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:11.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:11.115 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:11.115 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:11.116 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:11.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:11.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:11.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:11.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:11.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:11.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:11.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:11.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:11.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:11.738 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:11.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:11.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:11.739 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:12.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:12.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:12.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:12.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:12.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:12.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:12.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:12.611 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:12.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:12.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:12.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:12.612 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:12.613 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:13.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:13.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:13.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:13.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:13.144 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:13.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:13.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:13.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:13.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:13.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:13.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:13.246 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:13.247 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:13.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:13.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:13.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:13.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:14.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:14.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:14.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:14.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:14.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:14.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:14.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:14.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:14.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:14.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:14.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:14.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:14.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:14.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:14.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:14.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:14.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:14.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:14.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:14.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:14.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:14.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:15.106 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:15.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:15.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:15.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:15.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:15.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:15.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:15.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:15.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:15.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:15.619 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:15.620 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:16.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:16.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:16.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:16.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:16.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:16.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:16.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:16.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:16.236 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:16.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:16.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:16.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:16.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:16.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:16.605 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:16.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:16.606 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:17.137 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:17.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:17.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:17.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:17.138 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:17.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:17.139 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:17.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:17.139 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:17.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:17.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:17.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:17.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:17.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:17.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:17.740 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:17.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:17.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:17.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:17.741 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:17.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:17.742 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:18.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:18.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:18.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:18.107 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:18.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:18.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:18.612 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:18.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:18.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:18.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:18.613 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:18.613 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:18.614 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:19.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:19.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:19.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:19.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:19.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:19.219 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:19.220 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:19.220 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:19.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:19.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:19.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:19.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:19.224 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:19.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:19.590 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:19.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:19.591 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:20.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:20.093 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:20.093 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:20.093 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:20.093 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:20.094 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:20.094 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:20.094 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:20.094 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:20.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:20.865 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:20.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:20.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:20.866 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:20.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:20.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:20.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:20.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:20.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:20.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:20.869 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:20.870 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:21.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:21.264 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:21.264 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:21.264 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:21.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:21.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:21.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:21.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:21.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:21.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:21.759 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:21.759 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:21.759 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:22.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:22.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:22.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:22.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:22.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:22.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:22.385 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:22.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:22.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:22.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:22.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:22.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:22.386 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:22.760 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:22.760 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:22.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:22.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:23.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:23.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:23.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:23.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:23.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:23.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:23.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:23.294 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:23.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:23.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:23.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:23.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:23.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:23.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:23.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:23.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:23.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:23.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:23.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:23.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:23.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:23.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:24.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:24.258 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:24.258 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:24.258 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:24.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:24.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:24.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:24.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:24.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:24.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:24.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:24.758 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:24.758 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:25.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:25.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:25.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:25.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:25.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:25.398 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:25.399 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:25.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:25.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:25.399 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:25.400 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:25.400 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:25.400 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:25.761 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:25.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:25.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:25.761 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:26.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:26.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:26.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:26.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:26.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:26.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:26.292 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:26.293 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:26.293 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:26.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:26.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:26.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:26.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:26.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:26.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:26.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:26.957 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:26.957 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:26.957 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:27.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:27.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:27.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:27.327 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:27.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:27.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:27.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:27.829 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:27.829 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:27.829 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:28.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:28.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:28.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:28.443 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:28.444 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:28.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:28.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:28.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:28.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:28.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:28.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:28.546 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:28.546 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:28.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:28.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:28.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:28.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:29.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:29.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:29.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:29.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:29.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:29.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:29.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:29.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:29.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:30.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:30.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:30.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:30.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:30.011 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:30.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:30.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:30.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:30.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:30.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:30.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:30.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:30.113 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:30.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:30.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:30.478 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:30.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:30.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:30.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:30.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:30.999 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:31.000 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:31.000 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:31.000 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:31.000 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:31.001 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:31.546 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:31.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:31.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:31.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:31.547 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:31.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:31.648 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:31.648 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:31.648 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:31.649 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:32.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:32.014 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:32.014 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:32.014 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:32.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:32.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:32.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:32.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:32.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:32.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:32.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:32.524 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:32.524 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:33.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:33.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:33.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:33.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:33.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:33.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:33.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:33.128 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:33.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:33.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:33.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:33.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:33.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:33.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:33.499 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:33.499 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:33.500 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:34.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:34.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:34.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:34.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:34.003 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:34.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:34.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:34.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:34.004 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:34.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:34.499 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:34.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:34.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:34.500 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:34.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:34.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:34.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:34.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:34.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:34.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:34.601 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:34.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:34.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:34.969 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:34.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:34.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:35.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:35.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:35.470 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:35.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:35.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:35.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:35.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:35.471 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:35.472 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:36.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:36.004 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:36.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:36.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:36.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:36.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:36.107 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:36.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:36.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:36.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:36.108 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:36.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:36.109 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:36.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:36.480 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:36.481 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:36.481 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:36.994 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:36.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:36.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:36.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:36.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:36.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:36.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:36.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:36.997 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:37.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:37.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:37.494 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:37.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:37.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:37.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:37.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:37.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:37.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:37.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:37.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:37.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:37.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:37.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:37.971 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:37.971 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:37.972 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:38.476 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:38.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:38.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:38.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:38.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:38.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:38.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:38.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:38.479 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:39.001 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:39.001 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:39.001 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:39.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:39.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:39.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:39.105 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:39.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:39.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:39.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:39.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:39.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:39.106 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:39.465 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:39.465 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:39.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:39.466 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:39.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:39.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:39.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:39.968 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:39.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:39.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:39.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:39.969 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:39.970 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:40.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:01:40.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:01:40.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:40.479 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:40.480 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:40.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:40.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:40.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:40.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:40.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:40.583 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:40.583 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:40.584 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:40.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:40.958 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:40.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:40.959 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:41.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:41.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:41.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:41.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:41.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:41.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:41.493 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:41.494 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:41.495 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:42.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:01:42.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:01:42.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:42.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:42.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:42.186 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:42.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:42.188 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:42.188 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:42.188 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:42.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:42.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:42.189 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:42.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:42.570 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:42.571 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:42.571 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:43.068 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:43.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:43.069 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:43.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:43.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:43.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:43.070 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:43.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:43.071 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:43.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:01:43.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:01:43.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:43.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:43.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:43.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:43.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:43.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:43.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:43.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:43.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:43.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:43.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:44.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:44.082 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:44.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:44.084 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:44.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:44.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:44.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:44.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:44.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:45.175 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:01:45.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:01:45.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:45.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:45.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:45.277 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:45.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:45.278 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:45.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:45.279 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:45.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:45.280 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:45.281 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:45.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:45.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:45.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:45.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:46.145 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:46.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:46.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:46.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:46.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:46.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:46.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:46.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:46.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:46.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:01:46.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:01:46.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:46.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:46.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:46.756 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:46.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:46.757 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:46.758 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:46.759 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:46.759 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:46.759 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:46.760 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:47.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:47.126 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:47.126 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:47.127 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:47.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:47.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:47.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:47.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:47.628 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:47.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:47.629 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:47.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:47.629 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:48.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:48.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:48.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:48.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:48.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:48.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:48.257 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:48.258 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:48.258 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:48.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:48.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:48.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:48.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:48.621 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:48.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:48.622 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:49.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:49.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:49.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:49.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:49.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:49.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:49.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:49.149 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:49.150 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:49.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:49.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:49.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:49.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:49.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:49.779 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:49.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:49.780 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:49.781 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:49.781 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:49.781 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:49.781 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:49.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:50.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:50.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:50.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:50.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:50.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:50.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:50.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:50.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:50.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:50.666 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:50.666 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:50.666 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:50.666 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:51.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:51.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:51.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:51.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:51.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:51.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:51.295 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:51.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:51.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:51.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:51.296 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:51.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:51.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:51.673 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:51.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:51.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:51.674 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:52.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:52.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:52.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:52.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:52.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:52.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:52.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:52.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:52.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:52.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:52.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:52.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:52.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:52.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:52.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:52.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:52.769 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:52.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:52.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:52.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:52.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:52.771 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:53.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:53.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:53.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:53.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:53.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:53.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:53.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:53.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:53.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:53.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:53.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:53.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:53.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:54.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:54.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:54.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:54.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:54.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:54.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:54.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:54.285 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:54.285 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:54.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:54.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:54.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:54.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:54.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:54.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:54.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:54.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:55.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:55.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:55.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:55.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:55.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:55.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:55.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:55.156 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:55.156 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:01:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:01:55.677 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:55.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:55.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:55.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:55.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:55.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:55.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:55.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:55.779 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:55.780 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:55.780 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:56.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:56.146 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:56.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:56.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:56.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:56.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:56.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:56.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:56.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:56.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:56.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:56.656 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:56.656 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:57.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:57.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:57.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:57.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:57.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:57.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:57.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:57.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:57.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:57.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:57.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:57.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:57.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:57.643 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:57.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:57.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:57.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:58.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:58.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:58.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:58.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:58.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:58.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:58.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:58.149 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:58.149 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:01:58.674 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:01:58.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:01:58.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:01:58.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:01:58.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:01:58.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:01:58.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:01:58.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:01:58.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:01:58.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:01:58.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:01:58.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:01:58.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:01:59.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:01:59.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:01:59.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:01:59.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:01:59.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:01:59.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:01:59.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:01:59.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:01:59.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:01:59.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:01:59.634 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:01:59.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:01:59.635 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:00.142 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:02:00.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:02:00.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:00.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:00.143 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:00.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:00.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:00.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:00.244 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:00.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:00.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:00.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:00.245 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:00.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:00.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:00.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:00.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:01.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:01.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:01.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:01.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:01.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:01.134 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:01.135 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:01.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:01.135 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:01.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:02:01.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:02:01.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:01.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:01.655 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:01.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:01.755 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:01.755 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:01.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:01.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:01.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:01.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:01.756 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:02.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:02.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:02.124 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:02.125 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:02.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:02.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:02.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:02.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:02.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:02.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:02.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:02.626 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:02.627 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:03.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:03.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:03.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:03.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:03.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:03.221 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:03.222 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:03.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:03.222 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:03.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:03.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:03.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:03.223 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:03.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:03.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:03.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:03.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:04.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:04.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:04.072 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:04.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:04.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:04.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:04.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:04.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:04.074 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:04.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:04.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:04.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:04.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:04.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:04.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:04.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:04.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:04.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:04.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:04.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:04.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:04.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:05.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:05.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:05.055 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:05.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:05.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:05.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:05.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:05.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:05.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:05.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:05.627 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:05.627 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:05.628 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:06.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:06.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:06.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:06.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:06.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:06.251 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:06.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:06.252 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:06.252 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:06.253 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:06.253 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:06.253 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:06.253 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:06.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:06.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:06.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:06.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:07.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:07.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:07.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:07.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:07.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:07.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:07.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:07.196 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:07.196 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:07.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:02:07.712 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:02:07.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:07.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:07.713 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:07.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:07.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:07.815 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:07.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:07.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:07.817 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:07.818 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:07.818 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:08.203 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:08.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:08.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:08.205 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:08.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:08.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:08.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:08.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:08.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:08.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:08.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:08.718 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:08.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:09.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:09.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:09.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:09.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:09.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:09.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:09.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:09.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:09.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:09.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:09.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:09.415 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:09.415 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:09.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:09.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:09.788 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:09.789 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:10.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:10.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:10.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:10.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:10.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:10.306 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:10.307 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:10.307 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:10.307 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:10.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:10.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:10.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:10.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:10.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:10.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:10.915 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:10.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:10.916 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:10.916 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:10.916 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:10.917 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:10.917 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:11.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:11.283 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:11.283 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:11.284 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:11.794 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:11.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:11.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:11.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:11.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:11.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:11.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:11.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:11.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:12.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:12.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:12.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:12.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:12.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:12.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:12.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:12.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:12.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:12.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:12.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:12.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:12.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:12.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:12.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:12.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:12.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:13.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:13.374 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:13.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:13.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:13.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:13.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:13.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:13.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:13.376 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:13.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:13.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:13.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:13.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:13.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:13.979 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:13.979 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:13.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:13.979 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:13.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:13.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:13.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:13.980 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:14.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:14.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:14.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:14.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:14.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:14.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:14.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:14.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:14.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:14.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:14.833 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:14.834 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:14.834 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:15.381 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:15.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:15.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:15.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:15.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:15.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:15.482 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:15.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:15.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:15.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:15.483 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:15.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:15.484 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:15.847 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:15.848 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:15.848 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:15.848 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:16.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:16.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:16.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:16.344 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:16.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:16.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:16.345 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:16.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:16.346 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:16.863 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:02:16.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:02:16.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:16.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:16.864 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:16.964 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:16.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:16.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:16.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:16.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:16.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:16.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:16.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:17.384 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:17.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:17.384 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:17.385 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:17.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:17.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:17.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:17.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:17.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:17.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:17.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:17.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:17.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:18.414 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:02:18.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:02:18.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:18.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:18.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:18.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:18.516 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:18.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:18.516 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:18.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:18.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:18.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:18.517 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:18.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:18.881 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:18.881 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:18.881 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:19.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:19.375 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:19.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:19.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:19.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:19.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:19.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:19.377 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:19.377 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:19.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:19.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:19.912 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:19.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:19.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:20.012 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:20.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:20.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:20.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:20.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:20.013 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:20.014 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:20.014 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:20.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:20.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:20.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:20.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:20.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:20.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:20.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:20.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:20.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:20.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:20.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:20.876 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:20.876 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:21.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:21.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:21.386 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:21.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:21.387 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:21.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:21.487 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:21.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:21.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:21.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:21.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:21.488 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:21.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:21.862 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:21.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:21.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:21.863 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:22.360 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:22.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:22.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:22.361 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:22.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:22.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:22.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:22.362 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:22.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:23.019 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:23.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:23.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:23.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:23.020 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:23.120 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:23.121 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:23.121 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:23.121 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:23.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:23.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:23.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:23.122 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:23.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:23.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:23.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:23.582 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:24.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:24.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:24.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:24.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:24.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:24.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:24.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:24.082 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:24.082 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:24.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:24.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:24.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:24.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:24.585 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:24.685 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:24.686 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:24.686 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:24.686 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:24.686 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:24.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:24.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:24.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:25.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:25.056 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:25.057 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:25.057 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:25.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:25.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:25.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:25.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:25.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:25.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:25.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:25.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:25.559 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:26.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:26.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:26.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:26.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:26.082 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:26.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:26.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:26.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:26.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:26.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:26.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:26.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:26.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:26.548 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:26.548 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:26.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:26.549 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:27.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:27.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:27.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:27.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:27.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:27.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:27.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:27.057 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:27.057 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:27.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:27.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:27.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:27.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:27.559 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:27.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:27.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:27.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:27.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:27.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:27.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:27.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:27.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:28.027 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:28.027 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:28.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:28.028 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:28.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:28.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:28.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:28.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:28.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:28.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:28.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:28.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:29.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:02:29.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:02:29.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:29.051 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:29.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:29.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:29.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:29.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:29.154 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:29.154 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:29.154 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:29.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:29.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:29.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:29.528 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:29.528 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:29.528 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:30.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:30.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:30.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:30.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:30.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:30.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:30.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:30.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:30.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:30.552 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:02:30.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:02:30.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:30.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:30.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:30.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:30.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:30.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:30.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:30.655 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:30.655 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:30.655 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:30.655 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:31.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:31.058 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:31.059 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:31.059 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:31.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:31.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:31.565 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:31.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:31.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:31.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:32.128 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:32.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:32.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:32.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:32.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:32.230 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:32.230 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:32.230 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:32.231 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:32.231 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:32.231 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:32.231 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:32.232 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:32.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:32.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:32.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:32.602 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:33.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:33.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:33.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:33.103 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:33.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:33.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:33.104 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:33.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:33.105 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:33.608 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:33.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:33.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:33.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:33.609 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:33.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:33.711 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:33.711 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:33.712 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:33.712 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:33.712 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:33.712 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:33.713 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:34.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:34.079 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:34.080 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:34.080 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:34.600 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:34.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:34.601 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:34.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:34.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:34.605 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:34.607 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:34.609 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:34.610 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:35.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:35.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:35.089 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:35.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:35.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:35.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:35.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:35.191 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:35.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:35.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:35.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:35.193 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:35.193 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:35.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:35.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:35.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:35.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:36.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:36.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:36.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:36.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:36.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:36.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:36.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:36.059 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:36.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:36.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:36.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:36.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:36.681 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:36.682 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:36.780 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:36.781 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:36.781 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:36.781 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:36.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:36.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:36.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:36.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:37.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:37.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:37.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:37.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:37.649 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:37.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:37.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:37.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:37.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:37.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:37.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:37.652 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:37.654 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:38.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:38.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:38.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:38.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:38.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:38.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:38.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:38.291 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:38.291 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:38.291 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:38.291 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:38.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:38.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:38.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:38.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:38.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:38.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:39.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:39.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:39.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:39.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:39.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:39.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:39.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:39.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:39.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:39.671 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:39.671 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:39.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:39.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:39.772 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:39.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:39.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:39.773 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:39.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:39.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:39.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:40.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:40.133 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:40.133 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:40.134 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:40.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:40.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:40.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:40.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:40.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:40.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:40.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:40.627 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:40.627 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:41.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:41.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:41.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:41.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:41.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:41.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:41.270 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:41.271 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:41.271 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:41.271 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:41.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:41.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:41.272 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:41.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:41.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:41.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:41.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:42.123 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:42.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:42.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:42.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:42.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:42.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:42.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:42.125 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:42.125 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:42.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:42.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:42.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:42.618 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:42.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:42.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:42.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:42.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:42.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:42.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:42.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:42.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:42.721 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:43.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:43.086 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:43.086 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:43.086 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:43.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:43.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:43.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:43.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:43.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:43.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:43.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:43.579 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:43.579 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:44.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:02:44.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:02:44.108 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:44.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:44.109 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:44.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:44.209 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:44.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:44.209 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:44.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:44.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:44.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:44.210 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:44.571 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:44.571 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:44.572 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:44.572 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:45.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:45.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:45.080 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:45.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:45.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:45.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:45.081 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:45.081 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:45.082 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:45.590 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:02:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:02:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:45.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:45.691 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:45.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:45.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:45.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:45.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:45.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:45.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:45.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:46.067 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:46.067 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:46.067 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:46.068 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:46.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:46.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:46.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:46.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:46.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:46.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:46.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:46.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:46.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:47.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:47.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:47.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:47.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:47.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:47.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:47.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:47.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:47.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:47.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:47.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:47.275 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:47.275 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:47.646 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:47.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:47.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:47.647 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:48.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:48.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:48.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:48.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:48.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:48.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:48.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:48.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:48.155 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:48.675 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:48.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:48.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:48.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:48.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:48.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:48.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:48.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:48.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:48.777 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:48.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:48.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:48.778 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:49.165 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:49.166 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:49.166 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:49.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:49.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:49.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:49.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:49.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:49.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:49.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:49.659 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:49.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:49.659 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:50.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:02:50.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:02:50.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:50.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:50.276 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:50.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:50.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:50.377 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:50.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:50.378 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:50.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:50.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:50.379 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:50.746 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:50.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:50.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:50.747 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:51.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:51.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:51.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:51.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:51.265 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:51.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:51.266 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:51.266 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:51.266 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:51.784 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:02:51.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:02:51.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:51.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:51.785 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:51.888 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:51.888 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:51.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:51.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:51.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:51.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:51.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:51.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:52.267 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:52.267 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:52.267 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:52.268 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:52.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:52.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:52.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:52.768 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:52.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:52.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:52.769 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:52.769 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:52.770 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:53.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:02:53.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:02:53.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:53.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:53.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:53.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:53.415 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:53.415 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:53.416 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:53.416 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:53.416 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:53.417 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:53.417 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:53.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:53.793 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:53.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:53.794 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:54.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:54.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:54.309 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:54.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:54.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:54.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:54.310 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:54.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:54.311 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:54.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:02:54.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:02:54.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:54.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:54.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:55.031 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:55.031 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:55.031 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:55.031 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:55.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:55.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:55.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:55.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:55.394 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:55.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:55.394 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:55.395 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:55.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:55.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:55.906 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:55.907 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:55.907 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:55.907 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:55.907 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:55.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:55.908 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:56.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:02:56.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:02:56.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:56.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:56.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:56.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:56.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:56.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:56.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:56.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:56.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:56.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:56.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:56.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:56.955 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:56.955 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:56.955 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:57.488 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:57.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:57.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:57.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:57.489 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:57.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:57.490 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:57.493 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:57.494 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:58.016 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:02:58.016 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:02:58.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:58.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:58.017 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:58.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:58.119 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:58.119 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:58.119 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:58.119 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:58.120 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:58.120 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:58.120 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:02:58.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:02:58.495 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:02:58.495 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:02:58.496 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:02:59.036 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:02:59.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:02:59.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:02:59.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:02:59.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:02:59.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:02:59.236 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:02:59.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:02:59.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:02:59.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:02:59.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:02:59.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:02:59.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:02:59.542 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:02:59.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:02:59.642 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:02:59.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:02:59.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:02:59.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:02:59.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:02:59.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:02:59.644 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:00.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:00.010 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:00.011 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:00.011 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:00.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:00.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:00.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:00.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:00.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:00.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:00.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:00.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:00.523 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:01.045 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:01.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:01.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:01.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:01.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:01.146 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:01.147 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:01.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:01.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:01.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:01.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:01.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:01.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:01.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:01.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:01.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:01.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:02.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:02.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:02.023 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:02.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:02.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:02.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:02.024 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:02.025 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:02.025 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:02.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:02.538 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:02.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:02.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:02.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:02.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:02.638 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:02.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:02.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:02.638 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:02.641 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:02.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:02.643 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:03.030 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:03.030 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:03.031 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:03.031 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:03.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:03.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:03.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:03.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:03.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:03.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:03.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:03.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:03.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:04.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:03:04.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:03:04.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:04.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:04.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:04.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:04.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:04.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:04.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:04.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:04.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:04.183 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:04.186 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:04.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:04.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:04.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:04.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:05.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:05.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:05.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:05.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:05.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:05.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:05.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:05.059 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:05.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:05.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:03:05.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:03:05.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:05.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:05.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:05.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:05.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:05.688 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:05.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:05.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:05.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:05.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:05.690 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:06.141 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:06.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:06.142 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:06.143 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:06.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:06.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:06.640 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:06.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:06.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:06.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:06.641 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:06.642 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:06.642 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:07.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:07.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:07.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:07.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:07.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:07.258 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:07.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:07.259 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:07.260 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:07.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:07.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:07.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:07.261 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:07.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:07.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:07.623 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:07.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:08.116 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:08.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:08.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:08.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:08.117 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:08.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:08.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:08.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:08.118 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:08.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:08.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:08.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:08.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:08.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:08.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:08.762 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:08.762 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:08.762 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:08.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:08.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:08.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:08.763 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:09.129 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:09.129 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:09.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:09.130 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:09.716 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:09.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:09.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:09.717 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:09.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:09.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:09.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:09.718 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:09.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:10.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:03:10.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:03:10.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:10.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:10.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:10.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:10.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:10.312 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:10.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:10.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:10.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:10.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:10.314 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:10.676 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:10.676 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:10.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:10.677 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:11.171 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:11.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:11.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:11.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:11.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:11.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:11.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:11.173 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:11.173 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:11.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:03:11.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:03:11.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:11.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:11.737 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:11.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:11.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:11.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:11.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:11.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:11.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:11.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:11.840 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:12.198 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:12.198 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:12.198 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:12.198 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:12.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:12.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:12.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:12.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:12.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:12.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:12.689 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:12.689 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:12.690 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:13.201 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:13.201 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:13.201 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:13.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:13.202 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:13.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:13.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:13.304 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:13.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:13.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:13.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:13.305 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:13.306 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:13.668 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:13.668 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:13.669 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:13.669 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:14.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:14.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:14.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:14.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:14.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:14.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:14.168 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:14.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:14.169 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:14.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:14.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:14.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:14.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:14.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:14.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:14.800 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:14.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:14.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:14.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:14.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:14.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:14.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:15.164 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:15.165 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:15.165 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:15.165 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:15.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:15.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:15.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:15.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:15.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:15.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:15.665 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:15.665 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:15.665 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:16.196 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:16.196 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:16.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:16.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:16.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:16.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:16.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:16.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:16.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:16.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:16.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:16.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:16.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:16.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:16.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:16.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:16.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:17.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:17.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:17.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:17.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:17.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:17.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:17.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:17.158 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:17.158 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:17.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:03:17.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:03:17.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:17.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:17.813 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:17.913 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:17.914 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:17.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:17.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:17.914 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:17.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:17.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:17.915 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:18.275 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:18.275 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:18.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:18.276 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:18.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:18.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:18.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:18.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:18.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:18.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:18.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:18.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:18.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:19.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:19.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:19.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:19.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:19.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:19.422 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:19.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:19.423 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:19.423 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:19.423 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:19.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:19.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:19.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:19.784 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:19.784 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:19.785 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:19.785 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:20.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:20.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:20.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:20.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:20.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:20.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:20.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:20.284 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:20.285 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:20.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:20.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:20.792 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:20.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:20.793 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:20.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:20.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:20.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:20.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:20.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:20.894 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:20.895 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:20.895 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:21.258 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:21.258 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:21.259 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:21.259 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:21.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:21.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:21.787 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:21.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:21.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:21.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:21.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:21.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:21.791 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:22.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:22.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:22.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:22.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:22.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:22.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:22.416 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:22.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:22.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:22.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:22.418 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:22.419 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:22.419 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:22.795 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:22.796 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:22.796 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:22.796 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:23.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:23.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:23.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:23.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:23.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:23.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:23.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:23.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:23.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:23.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:23.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:23.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:23.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:23.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:24.036 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:24.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:24.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:24.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:24.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:24.037 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:24.038 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:24.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:24.400 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:24.400 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:24.401 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:24.401 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:24.922 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:24.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:24.927 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:24.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:24.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:24.928 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:24.929 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:24.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:24.929 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:25.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:25.591 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:25.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:25.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:25.592 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:25.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:25.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:25.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:25.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:25.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:25.693 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:25.694 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:25.694 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:26.060 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:26.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:26.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:26.061 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:26.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:26.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:26.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:26.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:26.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:26.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:26.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:26.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:26.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:27.111 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:03:27.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:03:27.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:27.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:27.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:27.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:27.213 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:27.213 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:27.213 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:27.213 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:27.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:27.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:27.214 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:27.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:27.586 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:27.588 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:27.589 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:28.084 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:28.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:28.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:28.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:28.085 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:28.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:28.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:28.086 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:28.086 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:28.630 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:03:28.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:03:28.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:28.631 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:28.632 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:28.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:28.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:28.732 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:28.732 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:28.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:28.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:28.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:28.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:29.102 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:29.103 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:29.103 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:29.104 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:29.635 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:29.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:29.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:29.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:29.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:29.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:29.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:29.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:29.637 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:30.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:30.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:30.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:30.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:30.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:30.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:30.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:30.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:30.297 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:30.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:30.298 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:30.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:30.299 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:30.664 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:30.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:30.665 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:30.665 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:31.170 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:31.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:31.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:31.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:31.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:31.174 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:31.174 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:31.174 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:31.175 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:31.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:31.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:31.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:31.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:31.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:31.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:31.799 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:31.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:31.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:31.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:31.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:31.800 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:31.801 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:32.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:32.166 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:32.166 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:32.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:32.656 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:32.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:32.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:32.657 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:32.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:32.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:32.658 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:32.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:32.658 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:33.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:33.211 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:33.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:33.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:33.212 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:33.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:33.324 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:33.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:33.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:33.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:33.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:33.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:33.325 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:33.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:33.681 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:33.681 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:33.681 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:34.182 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:34.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:34.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:34.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:34.183 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:34.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:34.184 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:34.184 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:34.185 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:34.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:34.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:34.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:34.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:34.723 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:34.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:34.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:34.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:34.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:34.824 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:34.825 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:34.825 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:34.825 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:35.200 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:35.201 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:35.202 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:35.204 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:35.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:35.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:35.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:35.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:35.697 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:35.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:35.698 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:35.699 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:35.699 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:36.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:36.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:36.263 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:36.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:36.264 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:36.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:36.366 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:36.367 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:36.367 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:36.367 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:36.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:36.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:36.368 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:36.773 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:36.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:36.774 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:36.782 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:37.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:37.302 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:37.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:37.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:37.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:37.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:37.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:37.304 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:37.304 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:37.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:37.857 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:37.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:37.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:37.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:37.969 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:37.971 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:37.973 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:37.973 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:37.973 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:37.973 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:37.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:37.974 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:38.331 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:38.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:38.332 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:38.332 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:38.983 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:38.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:38.984 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:38.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:38.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:38.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:38.986 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:38.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:38.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:39.532 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:03:39.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:03:39.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:39.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:39.533 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:39.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:39.633 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:39.633 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:39.634 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:39.634 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:39.634 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:39.634 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:39.634 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:40.002 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:40.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:40.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:40.015 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:40.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:40.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:40.506 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:40.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:40.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:40.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:40.507 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:40.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:40.508 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:41.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:41.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:41.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:41.078 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:41.079 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:41.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:41.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:41.179 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:41.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:41.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:41.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:41.180 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:41.181 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:41.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:41.557 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:41.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:41.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:42.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:42.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:42.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:42.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:42.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:42.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:42.050 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:42.051 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:42.051 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:42.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:42.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:42.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:42.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:42.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:42.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:42.684 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:42.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:42.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:42.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:42.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:42.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:42.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:43.046 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:43.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:43.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:43.047 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:43.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:43.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:43.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:43.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:43.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:43.596 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:43.597 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:43.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:43.597 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:44.123 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:44.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:44.124 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:44.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:44.125 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:44.224 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:44.225 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:44.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:44.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:44.226 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:44.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:44.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:44.227 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:44.595 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:44.595 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:44.595 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:44.596 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:45.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:45.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:45.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:45.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:45.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:45.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:45.092 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:45.092 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:45.093 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:45.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:45.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:45.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:45.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:45.621 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:45.718 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:45.719 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:45.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:45.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:45.719 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:45.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:45.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:45.720 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:46.086 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:46.087 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:46.087 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:46.087 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:46.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:46.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:46.720 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:46.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:46.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:46.721 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:46.722 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:46.722 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:46.722 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:47.278 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:47.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:47.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:47.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:47.279 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:47.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:47.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:47.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:47.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:47.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:47.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:47.381 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:47.382 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:47.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:47.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:47.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:47.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:48.252 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:48.253 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:48.253 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:48.253 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:48.253 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:48.254 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:48.254 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:48.254 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:48.254 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:48.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:48.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:48.844 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:48.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:48.845 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:48.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:48.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:48.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:48.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:48.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:48.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:48.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:48.946 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:49.312 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:49.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:49.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:49.313 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:49.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:49.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:49.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:49.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:49.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:49.816 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:49.816 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:49.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:49.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:50.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:50.351 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:50.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:50.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:50.352 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:50.448 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:50.449 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:50.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:50.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:50.449 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:50.450 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:50.450 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:50.450 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:50.818 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:50.818 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:50.819 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:50.819 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:51.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:51.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:51.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:51.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:51.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:51.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:51.326 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:51.327 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:51.327 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:51.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:51.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:51.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:51.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:51.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:51.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:51.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:51.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:51.964 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:51.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:51.965 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:51.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:51.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:52.330 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:52.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:52.330 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:52.331 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:52.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:52.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:52.837 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:52.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:52.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:52.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:52.838 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:52.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:52.839 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:53.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:03:53.353 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:03:53.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:53.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:53.354 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:53.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:53.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:53.455 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:53.455 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:53.455 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:53.456 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:53.456 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:53.456 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:53.822 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:53.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:53.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:53.823 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:54.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:54.338 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:54.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:54.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:54.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:54.339 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:54.340 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:54.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:54.340 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:54.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:03:54.834 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:03:54.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:54.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:54.835 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:54.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:54.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:54.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:54.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:54.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:54.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:54.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:54.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:55.343 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:55.343 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:55.344 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:55.344 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:55.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:55.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:55.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:55.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:55.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:55.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:55.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:55.842 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:55.843 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:56.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:03:56.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:03:56.477 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:56.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:56.478 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:56.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:56.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:56.580 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:56.580 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:56.580 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:56.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:56.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:56.581 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:57.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:57.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:57.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:57.033 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:57.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:57.556 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:57.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:57.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:57.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:57.557 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:57.558 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:57.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:57.558 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:58.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:03:58.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:03:58.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:58.090 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:58.091 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:58.191 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:58.192 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:58.192 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:58.198 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:58.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:58.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:58.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:58.199 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:03:58.555 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:03:58.555 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:03:58.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:03:58.556 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:03:59.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:03:59.047 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:03:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:03:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:03:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:03:59.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:03:59.049 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:03:59.049 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:03:59.049 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:03:59.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:03:59.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:03:59.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:03:59.570 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:03:59.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:03:59.670 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:03:59.671 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:03:59.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:03:59.671 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:03:59.672 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:03:59.672 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:03:59.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:03:59.673 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:00.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:00.043 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:00.043 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:00.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:00.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:00.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:00.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:00.678 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:00.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:00.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:00.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:00.680 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:00.680 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:01.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:04:01.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:04:01.240 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:01.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:01.241 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:01.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:01.341 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:01.341 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:01.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:01.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:01.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:01.342 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:01.343 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:01.709 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:01.709 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:01.709 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:01.710 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:02.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:02.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:02.284 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:02.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:02.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:02.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:02.285 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:02.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:02.286 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:02.775 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:04:02.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:04:02.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:02.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:02.776 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:02.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:02.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:02.935 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:02.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:02.937 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:02.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:02.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:02.938 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:03.237 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:03.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:03.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:03.237 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:03.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:03.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:03.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:03.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:03.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:03.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:03.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:03.745 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:03.745 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:04.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:04:04.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:04:04.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:04.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:04.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:04.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:04.424 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:04.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:04.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:04.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:04.424 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:04.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:04.425 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:04.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:04.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:04.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:04.791 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:05.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:05.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:05.349 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:05.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:05.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:05.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:05.350 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:05.351 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:05.351 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:05.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:04:05.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:04:05.854 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:05.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:05.855 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:05.954 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:05.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:05.955 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:05.955 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:05.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:05.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:05.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:05.956 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:06.336 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:06.337 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:06.337 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:06.337 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:06.825 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:06.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:06.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:06.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:06.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:06.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:06.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:06.827 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:06.828 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:07.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:04:07.380 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:04:07.382 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:07.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:07.383 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:07.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:07.474 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:07.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:07.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:07.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:07.475 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:07.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:07.476 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:07.867 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:07.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:07.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:07.868 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:08.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:08.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:08.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:08.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:08.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:08.364 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:08.365 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:08.365 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:08.365 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:08.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:04:08.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:04:08.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:08.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:08.933 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:09.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:09.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:09.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:09.035 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:09.413 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:09.413 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:09.413 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:09.414 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:09.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:09.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:09.916 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:09.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:09.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:09.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:09.917 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:09.918 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:09.918 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:10.471 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:04:10.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:04:10.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:10.472 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:10.473 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:10.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:10.573 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:10.573 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:10.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:10.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:10.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:10.574 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:10.575 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:10.944 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:10.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:10.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:10.945 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:11.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:11.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:11.446 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:11.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:11.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:11.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:11.447 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:11.447 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:11.448 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:12.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:04:12.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:04:12.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:12.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:12.013 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:12.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:12.110 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:12.110 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:12.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:12.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:12.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:12.111 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:12.112 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:12.498 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:12.499 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:12.499 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:12.499 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:13.006 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:13.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:13.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:13.007 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:13.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:13.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:13.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:13.008 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:13.008 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:13.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:04:13.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:04:13.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:13.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:13.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:13.623 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:13.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:13.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:13.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:13.624 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:13.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:13.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:13.625 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:13.985 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:13.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:13.985 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:13.986 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:14.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:14.495 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:14.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:14.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:14.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:14.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:14.496 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:14.497 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:14.498 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:15.073 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:04:15.074 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:04:15.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:15.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:15.075 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:15.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:15.169 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:15.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:15.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:15.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:15.170 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:15.171 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:15.171 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:15.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:15.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:15.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:15.538 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:16.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:16.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:16.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:16.040 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:16.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:16.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:16.041 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:16.041 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:16.042 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:16.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3822 2021-08-21T15:04:16.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3822 '!=' null 2021-08-21T15:04:16.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:16.588 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:16.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:16.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:16.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:16.690 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:16.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:16.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:16.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:16.691 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:16.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:17.071 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:17.072 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:17.072 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:17.072 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:17.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:17.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:17.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:17.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:17.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:17.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:17.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:17.567 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:17.568 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:18.187 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:04:18.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:04:18.188 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:18.189 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:18.189 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:18.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:18.286 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:18.287 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:18.287 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:18.287 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:18.287 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:18.287 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:18.288 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:18.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:18.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:18.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:18.664 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:19.165 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:19.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:19.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:19.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:19.166 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:19.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:19.167 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:19.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:19.167 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:19.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=3058 2021-08-21T15:04:19.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 3058 '!=' null 2021-08-21T15:04:19.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:19.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:19.688 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:19.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:19.789 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:19.789 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:19.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:19.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:19.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:19.790 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:19.791 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:20.156 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:20.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:20.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:20.157 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:20.660 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:20.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:20.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:20.661 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:20.662 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:20.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:20.663 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:20.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:20.663 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:21.175 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:04:21.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:04:21.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:21.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:21.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:21.272 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:21.273 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:21.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:21.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:21.273 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:21.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:21.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:21.274 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:21.639 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:21.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:21.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:21.639 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:22.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:22.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:22.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:22.172 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:22.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:22.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:22.173 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:22.173 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:22.174 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:22.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=5351 2021-08-21T15:04:22.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 5351 '!=' null 2021-08-21T15:04:22.777 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:22.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:22.778 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:22.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:22.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:22.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:22.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:22.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:22.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:22.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:22.881 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:23.248 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:23.248 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:23.248 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:23.249 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:23.747 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:23.748 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:23.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:23.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:23.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:23.749 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:23.750 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:23.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:23.750 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:24.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:04:24.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:04:24.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:24.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:24.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:24.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:24.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:24.362 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:24.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:24.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:24.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:24.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:24.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:24.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:24.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:24.746 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:24.747 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:25.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:25.259 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:25.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:25.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:25.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:25.260 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:25.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:25.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:25.262 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:25.757 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=4587 2021-08-21T15:04:25.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 4587 '!=' null 2021-08-21T15:04:25.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:25.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:25.758 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:25.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:25.859 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:25.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:25.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:25.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:25.860 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:25.861 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:25.862 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:26.228 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:26.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:26.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:26.228 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:26.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:26.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:26.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:26.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:26.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:26.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:26.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:26.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:26.733 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:27.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=6116 2021-08-21T15:04:27.261 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 6116 '!=' null 2021-08-21T15:04:27.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:27.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:27.262 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:27.362 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:27.363 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:27.363 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:27.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:27.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:27.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:27.364 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:27.365 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:27.727 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:27.727 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:27.728 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:27.728 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:28.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:28.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:28.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:28.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:28.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:28.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:28.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:28.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:28.437 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:28.930 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=2293 2021-08-21T15:04:28.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 2293 '!=' null 2021-08-21T15:04:28.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:28.931 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:28.932 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:29.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:29.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:29.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:29.033 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:29.033 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:29.033 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:29.033 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:29.034 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:29.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=0 2021-08-21T15:04:29.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:29.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:29.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:30.042 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 0 = 1 2021-08-21T15:04:30.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: test 0 '!=' 0 2021-08-21T15:04:30.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_is_making_recovery_progress 2021-08-21T15:04:30.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: get_is_making_recovery_progress: local recovery_progress 2021-08-21T15:04:30.043 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2021-08-21T15:04:30.044 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2021-08-21T15:04:30.044 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2021-08-21T15:04:30.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: ceph --format json status 2021-08-21T15:04:30.044 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2021-08-21T15:04:30.593 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: get_is_making_recovery_progress: local progress=2293 2021-08-21T15:04:30.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: get_is_making_recovery_progress: test 2293 '!=' null 2021-08-21T15:04:30.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1641: wait_for_clean: loop=0 2021-08-21T15:04:30.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: eval 2021-08-21T15:04:30.594 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: sleep 0.1 2021-08-21T15:04:30.695 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: loop+=1 2021-08-21T15:04:30.696 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: wait_for_clean: true 2021-08-21T15:04:30.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: get_num_active_clean 2021-08-21T15:04:30.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1342: get_num_active_clean: local expression 2021-08-21T15:04:30.696 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1343: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2021-08-21T15:04:30.697 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: get_num_active_clean: expression+='select(contains("stale") | not)' 2021-08-21T15:04:30.697 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1345: get_num_active_clean: ceph --format json pg dump pgs 2021-08-21T15:04:30.697 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2021-08-21T15:04:31.059 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: cur_active_clean=1 2021-08-21T15:04:31.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_num_pgs 2021-08-21T15:04:31.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: ceph --format json status 2021-08-21T15:04:31.060 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: get_num_pgs: jq .pgmap.num_pgs 2021-08-21T15:04:31.579 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: test 1 = 1 2021-08-21T15:04:31.868 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: break 2021-08-21T15:04:31.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: return 0 2021-08-21T15:04:31.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2021-08-21T15:04:31.869 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: flush_pg_stats: local timeout=300 2021-08-21T15:04:31.870 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ceph osd ls 2021-08-21T15:04:32.030 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2109: flush_pg_stats: ids='0 2021-08-21T15:04:32.030 INFO:tasks.workunit.client.0.smithi077.stderr:1 2021-08-21T15:04:32.030 INFO:tasks.workunit.client.0.smithi077.stderr:2 2021-08-21T15:04:32.031 INFO:tasks.workunit.client.0.smithi077.stderr:3 2021-08-21T15:04:32.031 INFO:tasks.workunit.client.0.smithi077.stderr:4 2021-08-21T15:04:32.031 INFO:tasks.workunit.client.0.smithi077.stderr:5' 2021-08-21T15:04:32.031 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: flush_pg_stats: seqs= 2021-08-21T15:04:32.032 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T15:04:32.032 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2021-08-21T15:04:32.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=21474836592 2021-08-21T15:04:32.160 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 21474836592 2021-08-21T15:04:32.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836592' 2021-08-21T15:04:32.161 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T15:04:32.161 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2021-08-21T15:04:32.290 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=42949673065 2021-08-21T15:04:32.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 42949673065 2021-08-21T15:04:32.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836592 1-42949673065' 2021-08-21T15:04:32.291 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T15:04:32.292 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2021-08-21T15:04:32.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=64424509539 2021-08-21T15:04:32.420 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 64424509539 2021-08-21T15:04:32.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539' 2021-08-21T15:04:32.421 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T15:04:32.421 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2021-08-21T15:04:32.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=85899346008 2021-08-21T15:04:32.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 85899346008 2021-08-21T15:04:32.553 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-85899346008' 2021-08-21T15:04:32.554 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T15:04:32.554 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2021-08-21T15:04:32.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=107374182491 2021-08-21T15:04:32.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 107374182491 2021-08-21T15:04:32.796 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-85899346008 4-107374182491' 2021-08-21T15:04:32.797 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2111: flush_pg_stats: for osd in $ids 2021-08-21T15:04:32.797 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2021-08-21T15:04:32.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2112: flush_pg_stats: seq=128849018958 2021-08-21T15:04:32.811 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: flush_pg_stats: test -z 128849018958 2021-08-21T15:04:32.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: flush_pg_stats: seqs=' 0-21474836592 1-42949673065 2-64424509539 3-85899346008 4-107374182491 5-128849018958' 2021-08-21T15:04:32.812 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T15:04:32.812 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 0-21474836592 2021-08-21T15:04:32.812 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T15:04:32.814 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=0 2021-08-21T15:04:32.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 0-21474836592 2021-08-21T15:04:32.814 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T15:04:32.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=21474836592 2021-08-21T15:04:32.815 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.0 seq 21474836592' 2021-08-21T15:04:32.815 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.0 seq 21474836592 2021-08-21T15:04:32.816 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 0 2021-08-21T15:04:33.193 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 21474836592 -lt 21474836592 2021-08-21T15:04:33.194 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T15:04:33.194 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 1-42949673065 2021-08-21T15:04:33.194 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T15:04:33.195 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=1 2021-08-21T15:04:33.195 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 1-42949673065 2021-08-21T15:04:33.196 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T15:04:33.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=42949673065 2021-08-21T15:04:33.197 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.1 seq 42949673065' 2021-08-21T15:04:33.198 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.1 seq 42949673065 2021-08-21T15:04:33.198 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 1 2021-08-21T15:04:33.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 42949673065 -lt 42949673065 2021-08-21T15:04:33.576 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T15:04:33.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 2-64424509539 2021-08-21T15:04:33.576 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T15:04:33.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=2 2021-08-21T15:04:33.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 2-64424509539 2021-08-21T15:04:33.577 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T15:04:33.577 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=64424509539 2021-08-21T15:04:33.578 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2021-08-21T15:04:33.578 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.2 seq 64424509539 2021-08-21T15:04:33.579 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 2 2021-08-21T15:04:33.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 64424509539 -lt 64424509539 2021-08-21T15:04:33.953 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T15:04:33.953 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 3-85899346008 2021-08-21T15:04:33.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T15:04:33.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=3 2021-08-21T15:04:33.961 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 3-85899346008 2021-08-21T15:04:33.961 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T15:04:33.962 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.3 seq 85899346008 2021-08-21T15:04:33.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=85899346008 2021-08-21T15:04:33.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.3 seq 85899346008' 2021-08-21T15:04:33.963 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 3 2021-08-21T15:04:34.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 85899346008 -lt 85899346008 2021-08-21T15:04:34.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T15:04:34.322 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 4-107374182491 2021-08-21T15:04:34.323 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T15:04:34.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=4 2021-08-21T15:04:34.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 4-107374182491 2021-08-21T15:04:34.324 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T15:04:34.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=107374182491 2021-08-21T15:04:34.325 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2021-08-21T15:04:34.326 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.4 seq 107374182491 2021-08-21T15:04:34.326 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T15:04:34.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182490 -lt 107374182491 2021-08-21T15:04:34.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: flush_pg_stats: sleep 1 2021-08-21T15:04:35.692 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2126: flush_pg_stats: '[' 300 -eq 0 ']' 2021-08-21T15:04:35.692 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 4 2021-08-21T15:04:36.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 107374182492 -lt 107374182491 2021-08-21T15:04:36.152 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: flush_pg_stats: for s in $seqs 2021-08-21T15:04:36.152 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: echo 5-128849018958 2021-08-21T15:04:36.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: cut -d - -f 1 2021-08-21T15:04:36.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: flush_pg_stats: osd=5 2021-08-21T15:04:36.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: echo 5-128849018958 2021-08-21T15:04:36.153 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: cut -d - -f 2 2021-08-21T15:04:36.153 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: flush_pg_stats: seq=128849018958 2021-08-21T15:04:36.154 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2123: flush_pg_stats: echo 'waiting osd.5 seq 128849018958' 2021-08-21T15:04:36.154 INFO:tasks.workunit.client.0.smithi077.stdout:waiting osd.5 seq 128849018958 2021-08-21T15:04:36.160 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: ceph osd last-stat-seq 5 2021-08-21T15:04:36.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: flush_pg_stats: test 128849018959 -lt 128849018958 2021-08-21T15:04:36.530 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2021-08-21T15:04:36.530 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2021-08-21T15:04:36.914 INFO:tasks.workunit.client.0.smithi077.stderr:dumped pgs 2021-08-21T15:04:36.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2021-08-21T15:04:36.935 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2021-08-21T15:04:36.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T15:04:36.936 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T15:04:36.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T15:04:36.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T15:04:36.936 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T15:04:37.232 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T15:04:37.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2021-08-21T15:04:37.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2021-08-21T15:04:37.233 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2021-08-21T15:04:37.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:37.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: _objectstore_tool_nodown: shift 2021-08-21T15:04:37.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: _objectstore_tool_nodown: local id=4 2021-08-21T15:04:37.234 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: _objectstore_tool_nodown: shift 2021-08-21T15:04:37.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2021-08-21T15:04:37.235 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2021-08-21T15:04:38.172 INFO:tasks.workunit.client.0.smithi077.stdout:{ 2021-08-21T15:04:38.172 INFO:tasks.workunit.client.0.smithi077.stdout: "pg_log_t": { 2021-08-21T15:04:38.172 INFO:tasks.workunit.client.0.smithi077.stdout: "head": "34'150", 2021-08-21T15:04:38.172 INFO:tasks.workunit.client.0.smithi077.stdout: "tail": "34'148", 2021-08-21T15:04:38.173 INFO:tasks.workunit.client.0.smithi077.stdout: "log": [ 2021-08-21T15:04:38.173 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.173 INFO:tasks.workunit.client.0.smithi077.stdout: "op": "modify", 2021-08-21T15:04:38.173 INFO:tasks.workunit.client.0.smithi077.stdout: "object": "1:e9c179f9:::obj-149:head", 2021-08-21T15:04:38.174 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'149", 2021-08-21T15:04:38.174 INFO:tasks.workunit.client.0.smithi077.stdout: "prior_version": "0'0", 2021-08-21T15:04:38.174 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4821.0:1", 2021-08-21T15:04:38.174 INFO:tasks.workunit.client.0.smithi077.stdout: "extra_reqids": [], 2021-08-21T15:04:38.174 INFO:tasks.workunit.client.0.smithi077.stdout: "mtime": "2021-08-21T14:59:28.364492+0000", 2021-08-21T15:04:38.175 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": 0, 2021-08-21T15:04:38.175 INFO:tasks.workunit.client.0.smithi077.stdout: "mod_desc": { 2021-08-21T15:04:38.175 INFO:tasks.workunit.client.0.smithi077.stdout: "object_mod_desc": { 2021-08-21T15:04:38.175 INFO:tasks.workunit.client.0.smithi077.stdout: "can_local_rollback": false, 2021-08-21T15:04:38.176 INFO:tasks.workunit.client.0.smithi077.stdout: "rollback_info_completed": false, 2021-08-21T15:04:38.176 INFO:tasks.workunit.client.0.smithi077.stdout: "ops": [] 2021-08-21T15:04:38.176 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.176 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.176 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_regions": { 2021-08-21T15:04:38.177 INFO:tasks.workunit.client.0.smithi077.stdout: "object_clean_regions": { 2021-08-21T15:04:38.177 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_offsets": "[9176~18446744073709542439]", 2021-08-21T15:04:38.177 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_omap": true, 2021-08-21T15:04:38.177 INFO:tasks.workunit.client.0.smithi077.stdout: "new_object": false 2021-08-21T15:04:38.177 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.178 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.178 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.178 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.178 INFO:tasks.workunit.client.0.smithi077.stdout: "op": "modify", 2021-08-21T15:04:38.179 INFO:tasks.workunit.client.0.smithi077.stdout: "object": "1:cb39fd60:::obj-150:head", 2021-08-21T15:04:38.179 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'150", 2021-08-21T15:04:38.179 INFO:tasks.workunit.client.0.smithi077.stdout: "prior_version": "0'0", 2021-08-21T15:04:38.179 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4824.0:1", 2021-08-21T15:04:38.180 INFO:tasks.workunit.client.0.smithi077.stdout: "extra_reqids": [], 2021-08-21T15:04:38.181 INFO:tasks.workunit.client.0.smithi077.stdout: "mtime": "2021-08-21T14:59:28.472878+0000", 2021-08-21T15:04:38.182 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": 0, 2021-08-21T15:04:38.182 INFO:tasks.workunit.client.0.smithi077.stdout: "mod_desc": { 2021-08-21T15:04:38.182 INFO:tasks.workunit.client.0.smithi077.stdout: "object_mod_desc": { 2021-08-21T15:04:38.183 INFO:tasks.workunit.client.0.smithi077.stdout: "can_local_rollback": false, 2021-08-21T15:04:38.183 INFO:tasks.workunit.client.0.smithi077.stdout: "rollback_info_completed": false, 2021-08-21T15:04:38.183 INFO:tasks.workunit.client.0.smithi077.stdout: "ops": [] 2021-08-21T15:04:38.184 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.184 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.184 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_regions": { 2021-08-21T15:04:38.185 INFO:tasks.workunit.client.0.smithi077.stdout: "object_clean_regions": { 2021-08-21T15:04:38.185 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_offsets": "[9176~18446744073709542439]", 2021-08-21T15:04:38.185 INFO:tasks.workunit.client.0.smithi077.stdout: "clean_omap": true, 2021-08-21T15:04:38.185 INFO:tasks.workunit.client.0.smithi077.stdout: "new_object": false 2021-08-21T15:04:38.186 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.186 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.186 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.186 INFO:tasks.workunit.client.0.smithi077.stdout: ], 2021-08-21T15:04:38.186 INFO:tasks.workunit.client.0.smithi077.stdout: "dups": [ 2021-08-21T15:04:38.187 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.187 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4377.0:1", 2021-08-21T15:04:38.187 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'1", 2021-08-21T15:04:38.188 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "1", 2021-08-21T15:04:38.188 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.188 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.188 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.189 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4380.0:1", 2021-08-21T15:04:38.189 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'2", 2021-08-21T15:04:38.189 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "2", 2021-08-21T15:04:38.189 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.189 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.190 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.190 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4383.0:1", 2021-08-21T15:04:38.190 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'3", 2021-08-21T15:04:38.190 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "3", 2021-08-21T15:04:38.191 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.191 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.192 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.193 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4386.0:1", 2021-08-21T15:04:38.193 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'4", 2021-08-21T15:04:38.193 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "4", 2021-08-21T15:04:38.194 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.194 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.194 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.194 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4389.0:1", 2021-08-21T15:04:38.195 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'5", 2021-08-21T15:04:38.195 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "5", 2021-08-21T15:04:38.195 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.195 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.196 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.196 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4392.0:1", 2021-08-21T15:04:38.196 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'6", 2021-08-21T15:04:38.196 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "6", 2021-08-21T15:04:38.197 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.197 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.197 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.197 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4395.0:1", 2021-08-21T15:04:38.198 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'7", 2021-08-21T15:04:38.198 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "7", 2021-08-21T15:04:38.198 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.198 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.199 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.199 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4398.0:1", 2021-08-21T15:04:38.199 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'8", 2021-08-21T15:04:38.200 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "8", 2021-08-21T15:04:38.200 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.200 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.200 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.201 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4401.0:1", 2021-08-21T15:04:38.201 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'9", 2021-08-21T15:04:38.201 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "9", 2021-08-21T15:04:38.201 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.201 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.202 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.202 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4404.0:1", 2021-08-21T15:04:38.203 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'10", 2021-08-21T15:04:38.203 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "10", 2021-08-21T15:04:38.203 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.203 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.204 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.204 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4407.0:1", 2021-08-21T15:04:38.204 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'11", 2021-08-21T15:04:38.204 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "11", 2021-08-21T15:04:38.205 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.205 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.205 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.205 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4410.0:1", 2021-08-21T15:04:38.206 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'12", 2021-08-21T15:04:38.206 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "12", 2021-08-21T15:04:38.206 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.207 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.207 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.208 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4413.0:1", 2021-08-21T15:04:38.208 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'13", 2021-08-21T15:04:38.208 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "13", 2021-08-21T15:04:38.208 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.209 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.209 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.209 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4416.0:1", 2021-08-21T15:04:38.209 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'14", 2021-08-21T15:04:38.209 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "14", 2021-08-21T15:04:38.210 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.210 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.210 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.210 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4419.0:1", 2021-08-21T15:04:38.211 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'15", 2021-08-21T15:04:38.211 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "15", 2021-08-21T15:04:38.211 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.211 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.211 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.212 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4422.0:1", 2021-08-21T15:04:38.212 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'16", 2021-08-21T15:04:38.212 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "16", 2021-08-21T15:04:38.212 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.213 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.213 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.213 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4425.0:1", 2021-08-21T15:04:38.213 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'17", 2021-08-21T15:04:38.213 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "17", 2021-08-21T15:04:38.214 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.214 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.214 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.214 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4428.0:1", 2021-08-21T15:04:38.215 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'18", 2021-08-21T15:04:38.215 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "18", 2021-08-21T15:04:38.215 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.215 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.215 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.216 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4431.0:1", 2021-08-21T15:04:38.216 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'19", 2021-08-21T15:04:38.216 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "19", 2021-08-21T15:04:38.216 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.217 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.217 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.217 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4434.0:1", 2021-08-21T15:04:38.217 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'20", 2021-08-21T15:04:38.217 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "20", 2021-08-21T15:04:38.218 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.218 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.218 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.218 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4437.0:1", 2021-08-21T15:04:38.219 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'21", 2021-08-21T15:04:38.219 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "21", 2021-08-21T15:04:38.219 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.219 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.220 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.224 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4440.0:1", 2021-08-21T15:04:38.225 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'22", 2021-08-21T15:04:38.225 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "22", 2021-08-21T15:04:38.225 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.225 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.226 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.226 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4443.0:1", 2021-08-21T15:04:38.226 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'23", 2021-08-21T15:04:38.226 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "23", 2021-08-21T15:04:38.227 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.227 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.227 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.227 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4446.0:1", 2021-08-21T15:04:38.228 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'24", 2021-08-21T15:04:38.228 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "24", 2021-08-21T15:04:38.228 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.228 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.229 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.229 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4449.0:1", 2021-08-21T15:04:38.229 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'25", 2021-08-21T15:04:38.229 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "25", 2021-08-21T15:04:38.229 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.230 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.230 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.230 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4452.0:1", 2021-08-21T15:04:38.230 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'26", 2021-08-21T15:04:38.231 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "26", 2021-08-21T15:04:38.231 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.231 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.231 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.231 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4455.0:1", 2021-08-21T15:04:38.232 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'27", 2021-08-21T15:04:38.232 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "27", 2021-08-21T15:04:38.232 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.232 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.233 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.233 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4458.0:1", 2021-08-21T15:04:38.233 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'28", 2021-08-21T15:04:38.233 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "28", 2021-08-21T15:04:38.233 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.234 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.234 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.234 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4461.0:1", 2021-08-21T15:04:38.234 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'29", 2021-08-21T15:04:38.235 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "29", 2021-08-21T15:04:38.235 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.235 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.235 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.236 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4464.0:1", 2021-08-21T15:04:38.236 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'30", 2021-08-21T15:04:38.236 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "30", 2021-08-21T15:04:38.236 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.236 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.237 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.237 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4467.0:1", 2021-08-21T15:04:38.237 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'31", 2021-08-21T15:04:38.237 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "31", 2021-08-21T15:04:38.238 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.238 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.238 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.239 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4470.0:1", 2021-08-21T15:04:38.239 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'32", 2021-08-21T15:04:38.239 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "32", 2021-08-21T15:04:38.239 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.240 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.240 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.240 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4473.0:1", 2021-08-21T15:04:38.240 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'33", 2021-08-21T15:04:38.240 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "33", 2021-08-21T15:04:38.241 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.241 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.241 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.241 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4476.0:1", 2021-08-21T15:04:38.242 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'34", 2021-08-21T15:04:38.242 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "34", 2021-08-21T15:04:38.242 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.242 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.242 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.243 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4479.0:1", 2021-08-21T15:04:38.243 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'35", 2021-08-21T15:04:38.243 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "35", 2021-08-21T15:04:38.243 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.244 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.244 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.244 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4482.0:1", 2021-08-21T15:04:38.244 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'36", 2021-08-21T15:04:38.244 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "36", 2021-08-21T15:04:38.245 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.245 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.245 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.246 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4485.0:1", 2021-08-21T15:04:38.246 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'37", 2021-08-21T15:04:38.246 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "37", 2021-08-21T15:04:38.246 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.247 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.247 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.247 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4488.0:1", 2021-08-21T15:04:38.248 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'38", 2021-08-21T15:04:38.248 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "38", 2021-08-21T15:04:38.248 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.248 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.248 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.249 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4491.0:1", 2021-08-21T15:04:38.249 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'39", 2021-08-21T15:04:38.249 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "39", 2021-08-21T15:04:38.249 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.250 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.250 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.250 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4494.0:1", 2021-08-21T15:04:38.250 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'40", 2021-08-21T15:04:38.250 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "40", 2021-08-21T15:04:38.251 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.251 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.251 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.251 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4497.0:1", 2021-08-21T15:04:38.252 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'41", 2021-08-21T15:04:38.252 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "41", 2021-08-21T15:04:38.252 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.252 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.252 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.253 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4500.0:1", 2021-08-21T15:04:38.253 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'42", 2021-08-21T15:04:38.253 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "42", 2021-08-21T15:04:38.253 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.253 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.254 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.254 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4503.0:1", 2021-08-21T15:04:38.254 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'43", 2021-08-21T15:04:38.254 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "43", 2021-08-21T15:04:38.255 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.255 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.255 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.255 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4506.0:1", 2021-08-21T15:04:38.256 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'44", 2021-08-21T15:04:38.256 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "44", 2021-08-21T15:04:38.256 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.256 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.256 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.257 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4509.0:1", 2021-08-21T15:04:38.257 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'45", 2021-08-21T15:04:38.257 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "45", 2021-08-21T15:04:38.257 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.258 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.258 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.258 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4512.0:1", 2021-08-21T15:04:38.261 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'46", 2021-08-21T15:04:38.261 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "46", 2021-08-21T15:04:38.262 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.262 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.262 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.262 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4515.0:1", 2021-08-21T15:04:38.263 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'47", 2021-08-21T15:04:38.263 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "47", 2021-08-21T15:04:38.263 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.263 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.264 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.264 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4518.0:1", 2021-08-21T15:04:38.264 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'48", 2021-08-21T15:04:38.264 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "48", 2021-08-21T15:04:38.264 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.265 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.265 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.265 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4521.0:1", 2021-08-21T15:04:38.265 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'49", 2021-08-21T15:04:38.266 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "49", 2021-08-21T15:04:38.266 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.266 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.266 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.267 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4524.0:1", 2021-08-21T15:04:38.267 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'50", 2021-08-21T15:04:38.267 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "50", 2021-08-21T15:04:38.267 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.267 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.268 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.268 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4527.0:1", 2021-08-21T15:04:38.268 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'51", 2021-08-21T15:04:38.268 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "51", 2021-08-21T15:04:38.269 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.269 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.269 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.269 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4530.0:1", 2021-08-21T15:04:38.270 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'52", 2021-08-21T15:04:38.270 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "52", 2021-08-21T15:04:38.270 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.270 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.271 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.271 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4533.0:1", 2021-08-21T15:04:38.271 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'53", 2021-08-21T15:04:38.271 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "53", 2021-08-21T15:04:38.271 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.272 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.272 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.272 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4536.0:1", 2021-08-21T15:04:38.272 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'54", 2021-08-21T15:04:38.273 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "54", 2021-08-21T15:04:38.273 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.273 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.273 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.273 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4539.0:1", 2021-08-21T15:04:38.274 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'55", 2021-08-21T15:04:38.274 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "55", 2021-08-21T15:04:38.274 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.274 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.275 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.275 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4542.0:1", 2021-08-21T15:04:38.275 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'56", 2021-08-21T15:04:38.275 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "56", 2021-08-21T15:04:38.276 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.276 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.276 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.276 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4545.0:1", 2021-08-21T15:04:38.277 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'57", 2021-08-21T15:04:38.277 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "57", 2021-08-21T15:04:38.277 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.277 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.278 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.278 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4548.0:1", 2021-08-21T15:04:38.278 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'58", 2021-08-21T15:04:38.278 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "58", 2021-08-21T15:04:38.279 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.279 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.279 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.279 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4551.0:1", 2021-08-21T15:04:38.280 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'59", 2021-08-21T15:04:38.280 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "59", 2021-08-21T15:04:38.280 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.280 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.281 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.281 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4554.0:1", 2021-08-21T15:04:38.281 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'60", 2021-08-21T15:04:38.281 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "60", 2021-08-21T15:04:38.282 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.282 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.282 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.282 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4557.0:1", 2021-08-21T15:04:38.283 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'61", 2021-08-21T15:04:38.283 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "61", 2021-08-21T15:04:38.283 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.283 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.284 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.284 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4560.0:1", 2021-08-21T15:04:38.284 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'62", 2021-08-21T15:04:38.284 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "62", 2021-08-21T15:04:38.284 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.285 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.285 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.285 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4563.0:1", 2021-08-21T15:04:38.285 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'63", 2021-08-21T15:04:38.286 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "63", 2021-08-21T15:04:38.286 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.286 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.286 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.287 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4566.0:1", 2021-08-21T15:04:38.287 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'64", 2021-08-21T15:04:38.287 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "64", 2021-08-21T15:04:38.287 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.287 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.288 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.288 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4569.0:1", 2021-08-21T15:04:38.288 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'65", 2021-08-21T15:04:38.288 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "65", 2021-08-21T15:04:38.289 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.289 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.289 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.289 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4572.0:1", 2021-08-21T15:04:38.290 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'66", 2021-08-21T15:04:38.290 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "66", 2021-08-21T15:04:38.290 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.290 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.291 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.291 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4575.0:1", 2021-08-21T15:04:38.291 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'67", 2021-08-21T15:04:38.291 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "67", 2021-08-21T15:04:38.291 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.292 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.292 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.292 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4578.0:1", 2021-08-21T15:04:38.292 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'68", 2021-08-21T15:04:38.293 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "68", 2021-08-21T15:04:38.293 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.293 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.293 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.293 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4581.0:1", 2021-08-21T15:04:38.294 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'69", 2021-08-21T15:04:38.294 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "69", 2021-08-21T15:04:38.294 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.294 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.295 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.295 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4584.0:1", 2021-08-21T15:04:38.295 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'70", 2021-08-21T15:04:38.295 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "70", 2021-08-21T15:04:38.296 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.296 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.296 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.296 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4587.0:1", 2021-08-21T15:04:38.296 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'71", 2021-08-21T15:04:38.297 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "71", 2021-08-21T15:04:38.297 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.297 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.297 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.298 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4590.0:1", 2021-08-21T15:04:38.298 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'72", 2021-08-21T15:04:38.298 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "72", 2021-08-21T15:04:38.298 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.299 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.299 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.299 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4593.0:1", 2021-08-21T15:04:38.299 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'73", 2021-08-21T15:04:38.300 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "73", 2021-08-21T15:04:38.300 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.300 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.300 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.300 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4596.0:1", 2021-08-21T15:04:38.301 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'74", 2021-08-21T15:04:38.301 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "74", 2021-08-21T15:04:38.301 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.301 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.302 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.302 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4599.0:1", 2021-08-21T15:04:38.302 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'75", 2021-08-21T15:04:38.303 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "75", 2021-08-21T15:04:38.303 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.303 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.303 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.304 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4602.0:1", 2021-08-21T15:04:38.304 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'76", 2021-08-21T15:04:38.304 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "76", 2021-08-21T15:04:38.304 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.305 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.305 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.305 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4605.0:1", 2021-08-21T15:04:38.305 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'77", 2021-08-21T15:04:38.305 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "77", 2021-08-21T15:04:38.306 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.306 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.306 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.306 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4608.0:1", 2021-08-21T15:04:38.307 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'78", 2021-08-21T15:04:38.307 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "78", 2021-08-21T15:04:38.307 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.308 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.308 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.308 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4611.0:1", 2021-08-21T15:04:38.308 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'79", 2021-08-21T15:04:38.309 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "79", 2021-08-21T15:04:38.309 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.309 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.309 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.310 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4614.0:1", 2021-08-21T15:04:38.310 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'80", 2021-08-21T15:04:38.310 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "80", 2021-08-21T15:04:38.310 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.311 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.311 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.311 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4617.0:1", 2021-08-21T15:04:38.311 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'81", 2021-08-21T15:04:38.312 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "81", 2021-08-21T15:04:38.312 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.312 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.312 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.312 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4620.0:1", 2021-08-21T15:04:38.313 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'82", 2021-08-21T15:04:38.313 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "82", 2021-08-21T15:04:38.313 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.313 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.314 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.314 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4623.0:1", 2021-08-21T15:04:38.314 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'83", 2021-08-21T15:04:38.314 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "83", 2021-08-21T15:04:38.315 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.315 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.315 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.315 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4626.0:1", 2021-08-21T15:04:38.315 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'84", 2021-08-21T15:04:38.316 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "84", 2021-08-21T15:04:38.316 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.316 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.316 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.317 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4629.0:1", 2021-08-21T15:04:38.317 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'85", 2021-08-21T15:04:38.317 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "85", 2021-08-21T15:04:38.317 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.318 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.318 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.318 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4632.0:1", 2021-08-21T15:04:38.318 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'86", 2021-08-21T15:04:38.319 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "86", 2021-08-21T15:04:38.319 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.319 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.319 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.320 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4635.0:1", 2021-08-21T15:04:38.320 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'87", 2021-08-21T15:04:38.320 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "87", 2021-08-21T15:04:38.320 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.321 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.321 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.321 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4638.0:1", 2021-08-21T15:04:38.321 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'88", 2021-08-21T15:04:38.321 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "88", 2021-08-21T15:04:38.322 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.322 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.322 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.322 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4641.0:1", 2021-08-21T15:04:38.323 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'89", 2021-08-21T15:04:38.323 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "89", 2021-08-21T15:04:38.323 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.323 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.324 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.324 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4644.0:1", 2021-08-21T15:04:38.324 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'90", 2021-08-21T15:04:38.324 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "90", 2021-08-21T15:04:38.324 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.325 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.325 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.325 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4647.0:1", 2021-08-21T15:04:38.325 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'91", 2021-08-21T15:04:38.326 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "91", 2021-08-21T15:04:38.326 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.326 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.326 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.327 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4650.0:1", 2021-08-21T15:04:38.327 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'92", 2021-08-21T15:04:38.327 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "92", 2021-08-21T15:04:38.327 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.327 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.328 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.328 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4653.0:1", 2021-08-21T15:04:38.328 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'93", 2021-08-21T15:04:38.328 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "93", 2021-08-21T15:04:38.329 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.329 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.329 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.329 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4656.0:1", 2021-08-21T15:04:38.330 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'94", 2021-08-21T15:04:38.330 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "94", 2021-08-21T15:04:38.330 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.330 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.330 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.331 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4659.0:1", 2021-08-21T15:04:38.331 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'95", 2021-08-21T15:04:38.331 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "95", 2021-08-21T15:04:38.331 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.332 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.332 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.332 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4662.0:1", 2021-08-21T15:04:38.332 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'96", 2021-08-21T15:04:38.333 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "96", 2021-08-21T15:04:38.333 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.333 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.333 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.333 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4665.0:1", 2021-08-21T15:04:38.334 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'97", 2021-08-21T15:04:38.334 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "97", 2021-08-21T15:04:38.334 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.334 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.335 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.335 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4668.0:1", 2021-08-21T15:04:38.335 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'98", 2021-08-21T15:04:38.335 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "98", 2021-08-21T15:04:38.336 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.336 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.336 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.336 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4671.0:1", 2021-08-21T15:04:38.337 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'99", 2021-08-21T15:04:38.337 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "99", 2021-08-21T15:04:38.337 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.337 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.337 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.338 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4674.0:1", 2021-08-21T15:04:38.338 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'100", 2021-08-21T15:04:38.338 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "100", 2021-08-21T15:04:38.338 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.339 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.339 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.339 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4677.0:1", 2021-08-21T15:04:38.339 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'101", 2021-08-21T15:04:38.340 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "101", 2021-08-21T15:04:38.340 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.340 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.340 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.341 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4680.0:1", 2021-08-21T15:04:38.341 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'102", 2021-08-21T15:04:38.341 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "102", 2021-08-21T15:04:38.341 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.342 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.342 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.342 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4683.0:1", 2021-08-21T15:04:38.342 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'103", 2021-08-21T15:04:38.342 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "103", 2021-08-21T15:04:38.343 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.343 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.343 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.343 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4686.0:1", 2021-08-21T15:04:38.344 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'104", 2021-08-21T15:04:38.344 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "104", 2021-08-21T15:04:38.344 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.344 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.344 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.345 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4689.0:1", 2021-08-21T15:04:38.345 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'105", 2021-08-21T15:04:38.345 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "105", 2021-08-21T15:04:38.345 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.346 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.346 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.346 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4692.0:1", 2021-08-21T15:04:38.346 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'106", 2021-08-21T15:04:38.347 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "106", 2021-08-21T15:04:38.347 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.347 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.347 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.347 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4695.0:1", 2021-08-21T15:04:38.348 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'107", 2021-08-21T15:04:38.348 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "107", 2021-08-21T15:04:38.348 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.348 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.349 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.349 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4698.0:1", 2021-08-21T15:04:38.349 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'108", 2021-08-21T15:04:38.349 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "108", 2021-08-21T15:04:38.350 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.350 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.350 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.350 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4701.0:1", 2021-08-21T15:04:38.350 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'109", 2021-08-21T15:04:38.351 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "109", 2021-08-21T15:04:38.351 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.351 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.351 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.352 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4704.0:1", 2021-08-21T15:04:38.352 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'110", 2021-08-21T15:04:38.352 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "110", 2021-08-21T15:04:38.352 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.352 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.353 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.353 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4707.0:1", 2021-08-21T15:04:38.353 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'111", 2021-08-21T15:04:38.353 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "111", 2021-08-21T15:04:38.354 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.354 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.354 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.354 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4710.0:1", 2021-08-21T15:04:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'112", 2021-08-21T15:04:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "112", 2021-08-21T15:04:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.355 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4713.0:1", 2021-08-21T15:04:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'113", 2021-08-21T15:04:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "113", 2021-08-21T15:04:38.356 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4716.0:1", 2021-08-21T15:04:38.357 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'114", 2021-08-21T15:04:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "114", 2021-08-21T15:04:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.358 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4719.0:1", 2021-08-21T15:04:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'115", 2021-08-21T15:04:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "115", 2021-08-21T15:04:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.359 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4722.0:1", 2021-08-21T15:04:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'116", 2021-08-21T15:04:38.360 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "116", 2021-08-21T15:04:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.361 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4725.0:1", 2021-08-21T15:04:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'117", 2021-08-21T15:04:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "117", 2021-08-21T15:04:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.362 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4728.0:1", 2021-08-21T15:04:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'118", 2021-08-21T15:04:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "118", 2021-08-21T15:04:38.363 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.364 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.364 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.364 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4731.0:1", 2021-08-21T15:04:38.364 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'119", 2021-08-21T15:04:38.365 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "119", 2021-08-21T15:04:38.365 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.365 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.365 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.365 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4734.0:1", 2021-08-21T15:04:38.366 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'120", 2021-08-21T15:04:38.366 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "120", 2021-08-21T15:04:38.366 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.366 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.367 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.367 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4737.0:1", 2021-08-21T15:04:38.367 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'121", 2021-08-21T15:04:38.367 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "121", 2021-08-21T15:04:38.368 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.368 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.368 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.368 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4740.0:1", 2021-08-21T15:04:38.368 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'122", 2021-08-21T15:04:38.369 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "122", 2021-08-21T15:04:38.369 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.369 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.369 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.370 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4743.0:1", 2021-08-21T15:04:38.370 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'123", 2021-08-21T15:04:38.370 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "123", 2021-08-21T15:04:38.370 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.371 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.371 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.371 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4746.0:1", 2021-08-21T15:04:38.371 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'124", 2021-08-21T15:04:38.371 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "124", 2021-08-21T15:04:38.372 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.372 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.372 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.372 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4749.0:1", 2021-08-21T15:04:38.373 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'125", 2021-08-21T15:04:38.373 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "125", 2021-08-21T15:04:38.373 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.373 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.374 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.374 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4752.0:1", 2021-08-21T15:04:38.374 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'126", 2021-08-21T15:04:38.374 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "126", 2021-08-21T15:04:38.374 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.375 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.375 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.375 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4755.0:1", 2021-08-21T15:04:38.375 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'127", 2021-08-21T15:04:38.376 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "127", 2021-08-21T15:04:38.376 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.376 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.376 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.377 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4758.0:1", 2021-08-21T15:04:38.377 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'128", 2021-08-21T15:04:38.377 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "128", 2021-08-21T15:04:38.377 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.377 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.378 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.378 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4761.0:1", 2021-08-21T15:04:38.378 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'129", 2021-08-21T15:04:38.378 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "129", 2021-08-21T15:04:38.379 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.379 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.379 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.379 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4764.0:1", 2021-08-21T15:04:38.379 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'130", 2021-08-21T15:04:38.380 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "130", 2021-08-21T15:04:38.380 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.380 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.381 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.381 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4767.0:1", 2021-08-21T15:04:38.381 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'131", 2021-08-21T15:04:38.381 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "131", 2021-08-21T15:04:38.382 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.382 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.382 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.382 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4770.0:1", 2021-08-21T15:04:38.383 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'132", 2021-08-21T15:04:38.383 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "132", 2021-08-21T15:04:38.383 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.383 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.383 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.384 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4773.0:1", 2021-08-21T15:04:38.384 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'133", 2021-08-21T15:04:38.384 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "133", 2021-08-21T15:04:38.384 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.385 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.385 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.385 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4776.0:1", 2021-08-21T15:04:38.385 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'134", 2021-08-21T15:04:38.386 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "134", 2021-08-21T15:04:38.386 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.386 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.386 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.387 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4779.0:1", 2021-08-21T15:04:38.387 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'135", 2021-08-21T15:04:38.387 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "135", 2021-08-21T15:04:38.387 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.388 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.388 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.388 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4782.0:1", 2021-08-21T15:04:38.388 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'136", 2021-08-21T15:04:38.389 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "136", 2021-08-21T15:04:38.389 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.389 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.389 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.390 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4785.0:1", 2021-08-21T15:04:38.390 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'137", 2021-08-21T15:04:38.390 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "137", 2021-08-21T15:04:38.390 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.390 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.391 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.391 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4788.0:1", 2021-08-21T15:04:38.391 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'138", 2021-08-21T15:04:38.391 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "138", 2021-08-21T15:04:38.392 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.392 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.392 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.392 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4791.0:1", 2021-08-21T15:04:38.393 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'139", 2021-08-21T15:04:38.393 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "139", 2021-08-21T15:04:38.393 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.393 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.394 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.394 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4794.0:1", 2021-08-21T15:04:38.394 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'140", 2021-08-21T15:04:38.394 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "140", 2021-08-21T15:04:38.395 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.395 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.395 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.395 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4797.0:1", 2021-08-21T15:04:38.396 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'141", 2021-08-21T15:04:38.396 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "141", 2021-08-21T15:04:38.396 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.396 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.396 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.397 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4800.0:1", 2021-08-21T15:04:38.397 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'142", 2021-08-21T15:04:38.397 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "142", 2021-08-21T15:04:38.397 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.398 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.398 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.398 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4803.0:1", 2021-08-21T15:04:38.398 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'143", 2021-08-21T15:04:38.399 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "143", 2021-08-21T15:04:38.399 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.399 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.399 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.400 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4806.0:1", 2021-08-21T15:04:38.400 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'144", 2021-08-21T15:04:38.400 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "144", 2021-08-21T15:04:38.400 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.401 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.401 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.401 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4809.0:1", 2021-08-21T15:04:38.401 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'145", 2021-08-21T15:04:38.401 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "145", 2021-08-21T15:04:38.402 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.402 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.402 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.403 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4812.0:1", 2021-08-21T15:04:38.403 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'146", 2021-08-21T15:04:38.403 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "146", 2021-08-21T15:04:38.403 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.404 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.404 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.404 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4815.0:1", 2021-08-21T15:04:38.404 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'147", 2021-08-21T15:04:38.680 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "147", 2021-08-21T15:04:38.681 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.681 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.681 INFO:tasks.workunit.client.0.smithi077.stdout: { 2021-08-21T15:04:38.682 INFO:tasks.workunit.client.0.smithi077.stdout: "reqid": "client.4818.0:1", 2021-08-21T15:04:38.682 INFO:tasks.workunit.client.0.smithi077.stdout: "version": "34'148", 2021-08-21T15:04:38.682 INFO:tasks.workunit.client.0.smithi077.stdout: "user_version": "148", 2021-08-21T15:04:38.682 INFO:tasks.workunit.client.0.smithi077.stdout: "return_code": "0" 2021-08-21T15:04:38.682 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.683 INFO:tasks.workunit.client.0.smithi077.stdout: ] 2021-08-21T15:04:38.683 INFO:tasks.workunit.client.0.smithi077.stdout: }, 2021-08-21T15:04:38.683 INFO:tasks.workunit.client.0.smithi077.stdout: "pg_missing_t": { 2021-08-21T15:04:38.683 INFO:tasks.workunit.client.0.smithi077.stdout: "missing": [], 2021-08-21T15:04:38.684 INFO:tasks.workunit.client.0.smithi077.stdout: "may_include_deletes": true 2021-08-21T15:04:38.684 INFO:tasks.workunit.client.0.smithi077.stdout: } 2021-08-21T15:04:38.685 INFO:tasks.workunit.client.0.smithi077.stdout:} 2021-08-21T15:04:38.686 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2021-08-21T15:04:38.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2021-08-21T15:04:38.687 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2021-08-21T15:04:38.687 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2021-08-21T15:04:38.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=148 2021-08-21T15:04:38.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: '[' 148 '!=' 148 ']' 2021-08-21T15:04:38.690 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2021-08-21T15:04:38.695 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to max 2 2021-08-21T15:04:38.696 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to copy log version 34'150 2021-08-21T15:04:38.696 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to copy log version 34'149 2021-08-21T15:04:38.696 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after earliest_dup_version 0 2021-08-21T15:04:38.696 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'1 2021-08-21T15:04:38.697 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'2 2021-08-21T15:04:38.697 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'3 2021-08-21T15:04:38.697 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'4 2021-08-21T15:04:38.697 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'5 2021-08-21T15:04:38.697 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'6 2021-08-21T15:04:38.698 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'7 2021-08-21T15:04:38.699 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'8 2021-08-21T15:04:38.699 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'9 2021-08-21T15:04:38.699 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'10 2021-08-21T15:04:38.699 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'11 2021-08-21T15:04:38.699 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'12 2021-08-21T15:04:38.700 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'13 2021-08-21T15:04:38.700 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'14 2021-08-21T15:04:38.700 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'15 2021-08-21T15:04:38.701 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'16 2021-08-21T15:04:38.701 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'17 2021-08-21T15:04:38.701 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'18 2021-08-21T15:04:38.701 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'19 2021-08-21T15:04:38.702 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'20 2021-08-21T15:04:38.702 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'21 2021-08-21T15:04:38.702 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'22 2021-08-21T15:04:38.702 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'23 2021-08-21T15:04:38.703 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'24 2021-08-21T15:04:38.703 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'25 2021-08-21T15:04:38.703 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'26 2021-08-21T15:04:38.703 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'27 2021-08-21T15:04:38.704 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'28 2021-08-21T15:04:38.705 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'29 2021-08-21T15:04:38.705 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'30 2021-08-21T15:04:38.705 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'31 2021-08-21T15:04:38.705 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'32 2021-08-21T15:04:38.706 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'33 2021-08-21T15:04:38.706 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'34 2021-08-21T15:04:38.706 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'35 2021-08-21T15:04:38.706 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'36 2021-08-21T15:04:38.707 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'37 2021-08-21T15:04:38.707 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'38 2021-08-21T15:04:38.707 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'39 2021-08-21T15:04:38.707 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'40 2021-08-21T15:04:38.708 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'41 2021-08-21T15:04:38.708 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'42 2021-08-21T15:04:38.708 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'43 2021-08-21T15:04:38.708 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'44 2021-08-21T15:04:38.709 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'45 2021-08-21T15:04:38.709 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'46 2021-08-21T15:04:38.710 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'47 2021-08-21T15:04:38.710 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'48 2021-08-21T15:04:38.710 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'49 2021-08-21T15:04:38.710 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'50 2021-08-21T15:04:38.710 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'51 2021-08-21T15:04:38.711 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'52 2021-08-21T15:04:38.711 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'53 2021-08-21T15:04:38.711 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'54 2021-08-21T15:04:38.711 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'55 2021-08-21T15:04:38.712 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'56 2021-08-21T15:04:38.712 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'57 2021-08-21T15:04:38.712 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'58 2021-08-21T15:04:38.712 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'59 2021-08-21T15:04:38.713 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'60 2021-08-21T15:04:38.713 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'61 2021-08-21T15:04:38.713 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'62 2021-08-21T15:04:38.713 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'63 2021-08-21T15:04:38.713 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'64 2021-08-21T15:04:38.714 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'65 2021-08-21T15:04:38.714 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'66 2021-08-21T15:04:38.714 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'67 2021-08-21T15:04:38.715 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'68 2021-08-21T15:04:38.716 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'69 2021-08-21T15:04:38.716 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'70 2021-08-21T15:04:38.716 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'71 2021-08-21T15:04:38.716 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'72 2021-08-21T15:04:38.717 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'73 2021-08-21T15:04:38.717 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'74 2021-08-21T15:04:38.717 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'75 2021-08-21T15:04:38.717 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'76 2021-08-21T15:04:38.717 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'77 2021-08-21T15:04:38.718 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'78 2021-08-21T15:04:38.718 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'79 2021-08-21T15:04:38.718 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'80 2021-08-21T15:04:38.718 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'81 2021-08-21T15:04:38.719 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'82 2021-08-21T15:04:38.719 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'83 2021-08-21T15:04:38.719 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'84 2021-08-21T15:04:38.720 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'85 2021-08-21T15:04:38.720 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'86 2021-08-21T15:04:38.720 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'87 2021-08-21T15:04:38.720 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'88 2021-08-21T15:04:38.721 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'89 2021-08-21T15:04:38.721 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'90 2021-08-21T15:04:38.721 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'91 2021-08-21T15:04:38.721 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'92 2021-08-21T15:04:38.722 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'93 2021-08-21T15:04:38.722 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'94 2021-08-21T15:04:38.722 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'95 2021-08-21T15:04:38.723 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'96 2021-08-21T15:04:38.723 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'97 2021-08-21T15:04:38.723 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'98 2021-08-21T15:04:38.723 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'99 2021-08-21T15:04:38.723 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'100 2021-08-21T15:04:38.724 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-08-21T15:04:38.724 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-08-21T15:04:38.724 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-08-21T15:04:38.724 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-08-21T15:04:38.725 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-08-21T15:04:38.726 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-08-21T15:04:38.726 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-08-21T15:04:38.726 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-08-21T15:04:38.726 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-08-21T15:04:38.727 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-08-21T15:04:38.727 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-08-21T15:04:38.727 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-08-21T15:04:38.727 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-08-21T15:04:38.728 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-08-21T15:04:38.728 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-08-21T15:04:38.728 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-08-21T15:04:38.728 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-08-21T15:04:38.729 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-08-21T15:04:38.729 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-08-21T15:04:38.729 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-08-21T15:04:38.729 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-08-21T15:04:38.730 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-08-21T15:04:38.730 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-08-21T15:04:38.730 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-08-21T15:04:38.731 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-08-21T15:04:38.731 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-08-21T15:04:38.731 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-08-21T15:04:38.731 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-08-21T15:04:38.732 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-08-21T15:04:38.732 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-08-21T15:04:38.732 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-08-21T15:04:38.732 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-08-21T15:04:38.733 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-08-21T15:04:38.733 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-08-21T15:04:38.733 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-08-21T15:04:38.733 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-08-21T15:04:38.733 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-08-21T15:04:38.734 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-08-21T15:04:38.734 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-08-21T15:04:38.734 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-08-21T15:04:38.734 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T15:04:38.735 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T15:04:38.735 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T15:04:38.735 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T15:04:38.736 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T15:04:38.737 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T15:04:38.737 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T15:04:38.737 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.916+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T15:04:38.738 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to max 2 2021-08-21T15:04:38.738 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to copy log version 34'150 2021-08-21T15:04:38.738 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to copy log version 34'149 2021-08-21T15:04:38.739 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after earliest_dup_version 0 2021-08-21T15:04:38.739 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'1 2021-08-21T15:04:38.739 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'2 2021-08-21T15:04:38.739 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'3 2021-08-21T15:04:38.740 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'4 2021-08-21T15:04:38.740 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'5 2021-08-21T15:04:38.741 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'6 2021-08-21T15:04:38.741 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'7 2021-08-21T15:04:38.741 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'8 2021-08-21T15:04:38.742 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'9 2021-08-21T15:04:38.742 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'10 2021-08-21T15:04:38.742 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'11 2021-08-21T15:04:38.742 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'12 2021-08-21T15:04:38.743 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'13 2021-08-21T15:04:38.743 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'14 2021-08-21T15:04:38.743 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'15 2021-08-21T15:04:38.743 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'16 2021-08-21T15:04:38.744 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'17 2021-08-21T15:04:38.744 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'18 2021-08-21T15:04:38.744 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'19 2021-08-21T15:04:38.744 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'20 2021-08-21T15:04:38.744 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'21 2021-08-21T15:04:38.745 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'22 2021-08-21T15:04:38.745 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'23 2021-08-21T15:04:38.745 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'24 2021-08-21T15:04:38.745 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'25 2021-08-21T15:04:38.745 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'26 2021-08-21T15:04:38.747 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'27 2021-08-21T15:04:38.747 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'28 2021-08-21T15:04:38.747 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'29 2021-08-21T15:04:38.747 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'30 2021-08-21T15:04:38.748 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'31 2021-08-21T15:04:38.748 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'32 2021-08-21T15:04:38.748 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'33 2021-08-21T15:04:38.748 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'34 2021-08-21T15:04:38.749 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'35 2021-08-21T15:04:38.749 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'36 2021-08-21T15:04:38.749 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'37 2021-08-21T15:04:38.749 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'38 2021-08-21T15:04:38.750 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'39 2021-08-21T15:04:38.750 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'40 2021-08-21T15:04:38.750 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'41 2021-08-21T15:04:38.750 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'42 2021-08-21T15:04:38.751 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'43 2021-08-21T15:04:38.751 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'44 2021-08-21T15:04:38.751 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'45 2021-08-21T15:04:38.752 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'46 2021-08-21T15:04:38.752 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'47 2021-08-21T15:04:38.752 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'48 2021-08-21T15:04:38.752 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'49 2021-08-21T15:04:38.753 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'50 2021-08-21T15:04:38.753 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'51 2021-08-21T15:04:38.753 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'52 2021-08-21T15:04:38.753 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'53 2021-08-21T15:04:38.754 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'54 2021-08-21T15:04:38.754 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'55 2021-08-21T15:04:38.754 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'56 2021-08-21T15:04:38.754 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'57 2021-08-21T15:04:38.754 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'58 2021-08-21T15:04:38.755 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'59 2021-08-21T15:04:38.755 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'60 2021-08-21T15:04:38.755 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'61 2021-08-21T15:04:38.756 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'62 2021-08-21T15:04:38.756 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'63 2021-08-21T15:04:38.756 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'64 2021-08-21T15:04:38.757 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'65 2021-08-21T15:04:38.757 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'66 2021-08-21T15:04:38.758 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'67 2021-08-21T15:04:38.758 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'68 2021-08-21T15:04:38.758 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'69 2021-08-21T15:04:38.758 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'70 2021-08-21T15:04:38.759 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'71 2021-08-21T15:04:38.759 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'72 2021-08-21T15:04:38.759 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'73 2021-08-21T15:04:38.759 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'74 2021-08-21T15:04:38.760 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'75 2021-08-21T15:04:38.760 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'76 2021-08-21T15:04:38.760 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'77 2021-08-21T15:04:38.760 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'78 2021-08-21T15:04:38.760 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'79 2021-08-21T15:04:38.761 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'80 2021-08-21T15:04:38.761 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'81 2021-08-21T15:04:38.762 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'82 2021-08-21T15:04:38.762 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'83 2021-08-21T15:04:38.762 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'84 2021-08-21T15:04:38.762 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'85 2021-08-21T15:04:38.763 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'86 2021-08-21T15:04:38.763 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'87 2021-08-21T15:04:38.763 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'88 2021-08-21T15:04:38.763 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'89 2021-08-21T15:04:38.763 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'90 2021-08-21T15:04:38.764 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'91 2021-08-21T15:04:38.764 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'92 2021-08-21T15:04:38.764 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'93 2021-08-21T15:04:38.764 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'94 2021-08-21T15:04:38.765 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'95 2021-08-21T15:04:38.765 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'96 2021-08-21T15:04:38.765 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'97 2021-08-21T15:04:38.765 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'98 2021-08-21T15:04:38.766 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'99 2021-08-21T15:04:38.766 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'100 2021-08-21T15:04:38.766 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-08-21T15:04:38.766 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-08-21T15:04:38.766 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-08-21T15:04:38.767 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-08-21T15:04:38.768 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-08-21T15:04:38.768 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-08-21T15:04:38.768 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-08-21T15:04:38.769 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-08-21T15:04:38.769 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-08-21T15:04:38.769 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-08-21T15:04:38.769 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-08-21T15:04:38.770 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-08-21T15:04:38.770 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-08-21T15:04:38.770 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-08-21T15:04:38.770 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-08-21T15:04:38.771 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-08-21T15:04:38.771 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-08-21T15:04:38.771 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-08-21T15:04:38.771 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-08-21T15:04:38.772 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-08-21T15:04:38.772 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-08-21T15:04:38.773 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-08-21T15:04:38.773 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-08-21T15:04:38.773 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-08-21T15:04:38.774 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-08-21T15:04:38.774 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-08-21T15:04:38.775 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-08-21T15:04:38.775 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-08-21T15:04:38.775 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-08-21T15:04:38.775 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-08-21T15:04:38.775 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-08-21T15:04:38.776 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-08-21T15:04:38.776 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-08-21T15:04:38.776 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-08-21T15:04:38.776 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-08-21T15:04:38.777 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-08-21T15:04:38.777 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-08-21T15:04:38.777 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-08-21T15:04:38.777 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-08-21T15:04:38.777 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-08-21T15:04:38.778 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T15:04:38.778 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T15:04:38.778 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T15:04:38.778 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T15:04:38.779 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T15:04:38.779 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T15:04:38.779 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T15:04:38.779 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T15:04:38.780 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to max 2 2021-08-21T15:04:38.780 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to copy log version 34'150 2021-08-21T15:04:38.780 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to copy log version 34'149 2021-08-21T15:04:38.780 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after earliest_dup_version 0 2021-08-21T15:04:38.780 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'1 2021-08-21T15:04:38.781 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'2 2021-08-21T15:04:38.781 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'3 2021-08-21T15:04:38.783 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'4 2021-08-21T15:04:38.783 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'5 2021-08-21T15:04:38.784 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'6 2021-08-21T15:04:38.784 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'7 2021-08-21T15:04:38.784 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'8 2021-08-21T15:04:38.784 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'9 2021-08-21T15:04:38.785 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'10 2021-08-21T15:04:38.785 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'11 2021-08-21T15:04:38.785 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'12 2021-08-21T15:04:38.785 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'13 2021-08-21T15:04:38.786 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'14 2021-08-21T15:04:38.787 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'15 2021-08-21T15:04:38.787 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'16 2021-08-21T15:04:38.787 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'17 2021-08-21T15:04:38.788 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'18 2021-08-21T15:04:38.788 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'19 2021-08-21T15:04:38.789 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'20 2021-08-21T15:04:38.789 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'21 2021-08-21T15:04:38.789 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'22 2021-08-21T15:04:38.789 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'23 2021-08-21T15:04:38.790 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'24 2021-08-21T15:04:38.790 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'25 2021-08-21T15:04:38.790 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'26 2021-08-21T15:04:38.790 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'27 2021-08-21T15:04:38.791 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'28 2021-08-21T15:04:38.791 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'29 2021-08-21T15:04:38.791 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'30 2021-08-21T15:04:38.792 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'31 2021-08-21T15:04:38.792 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'32 2021-08-21T15:04:38.792 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'33 2021-08-21T15:04:38.793 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'34 2021-08-21T15:04:38.793 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'35 2021-08-21T15:04:38.793 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'36 2021-08-21T15:04:38.794 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'37 2021-08-21T15:04:38.794 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'38 2021-08-21T15:04:38.794 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'39 2021-08-21T15:04:38.795 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'40 2021-08-21T15:04:38.795 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'41 2021-08-21T15:04:38.795 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'42 2021-08-21T15:04:38.796 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'43 2021-08-21T15:04:38.796 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'44 2021-08-21T15:04:38.796 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'45 2021-08-21T15:04:38.797 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'46 2021-08-21T15:04:38.798 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'47 2021-08-21T15:04:38.798 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'48 2021-08-21T15:04:38.799 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'49 2021-08-21T15:04:38.799 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'50 2021-08-21T15:04:38.799 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'51 2021-08-21T15:04:38.799 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'52 2021-08-21T15:04:38.800 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'53 2021-08-21T15:04:38.800 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'54 2021-08-21T15:04:38.800 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'55 2021-08-21T15:04:38.800 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'56 2021-08-21T15:04:38.801 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'57 2021-08-21T15:04:38.801 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'58 2021-08-21T15:04:38.801 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'59 2021-08-21T15:04:38.801 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'60 2021-08-21T15:04:38.801 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'61 2021-08-21T15:04:38.802 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'62 2021-08-21T15:04:38.802 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'63 2021-08-21T15:04:38.802 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'64 2021-08-21T15:04:38.802 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'65 2021-08-21T15:04:38.803 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'66 2021-08-21T15:04:38.803 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'67 2021-08-21T15:04:38.803 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'68 2021-08-21T15:04:38.803 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'69 2021-08-21T15:04:38.804 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'70 2021-08-21T15:04:38.804 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'71 2021-08-21T15:04:38.804 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'72 2021-08-21T15:04:38.804 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'73 2021-08-21T15:04:38.804 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'74 2021-08-21T15:04:38.805 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'75 2021-08-21T15:04:38.805 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'76 2021-08-21T15:04:38.805 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'77 2021-08-21T15:04:38.805 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'78 2021-08-21T15:04:38.806 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'79 2021-08-21T15:04:38.806 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'80 2021-08-21T15:04:38.806 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'81 2021-08-21T15:04:38.806 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'82 2021-08-21T15:04:38.806 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'83 2021-08-21T15:04:38.807 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'84 2021-08-21T15:04:38.807 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'85 2021-08-21T15:04:38.807 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'86 2021-08-21T15:04:38.807 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'87 2021-08-21T15:04:38.808 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'88 2021-08-21T15:04:38.808 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'89 2021-08-21T15:04:38.808 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'90 2021-08-21T15:04:38.808 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'91 2021-08-21T15:04:38.808 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'92 2021-08-21T15:04:38.809 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'93 2021-08-21T15:04:38.809 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'94 2021-08-21T15:04:38.809 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'95 2021-08-21T15:04:38.809 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'96 2021-08-21T15:04:38.809 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'97 2021-08-21T15:04:38.810 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'98 2021-08-21T15:04:38.810 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'99 2021-08-21T15:04:38.810 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'100 2021-08-21T15:04:38.810 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-08-21T15:04:38.811 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-08-21T15:04:38.811 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-08-21T15:04:38.811 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-08-21T15:04:38.811 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-08-21T15:04:38.811 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-08-21T15:04:38.812 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-08-21T15:04:38.812 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-08-21T15:04:38.812 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-08-21T15:04:38.812 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-08-21T15:04:38.813 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-08-21T15:04:38.813 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-08-21T15:04:38.813 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-08-21T15:04:38.813 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-08-21T15:04:38.813 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-08-21T15:04:38.814 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-08-21T15:04:38.814 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-08-21T15:04:38.814 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-08-21T15:04:38.814 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-08-21T15:04:38.815 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-08-21T15:04:38.815 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-08-21T15:04:38.815 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-08-21T15:04:38.815 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-08-21T15:04:38.815 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-08-21T15:04:38.816 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-08-21T15:04:38.816 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-08-21T15:04:38.816 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-08-21T15:04:38.816 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-08-21T15:04:38.817 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-08-21T15:04:38.818 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-08-21T15:04:38.818 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-08-21T15:04:38.818 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-08-21T15:04:38.818 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-08-21T15:04:38.819 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-08-21T15:04:38.819 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-08-21T15:04:38.819 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-08-21T15:04:38.819 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-08-21T15:04:38.820 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-08-21T15:04:38.820 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-08-21T15:04:38.820 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-08-21T15:04:38.820 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T15:04:38.820 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T15:04:38.821 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T15:04:38.821 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T15:04:38.821 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T15:04:38.821 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T15:04:38.822 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T15:04:38.822 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T14:59:30.917+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T15:04:38.822 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_after v 34'150 2021-08-21T15:04:38.822 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after earliest_dup_version 0 2021-08-21T15:04:38.822 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'1 2021-08-21T15:04:38.823 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'2 2021-08-21T15:04:38.823 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'3 2021-08-21T15:04:38.823 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'4 2021-08-21T15:04:38.823 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'5 2021-08-21T15:04:38.824 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'6 2021-08-21T15:04:38.824 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'7 2021-08-21T15:04:38.824 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'8 2021-08-21T15:04:38.824 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'9 2021-08-21T15:04:38.824 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'10 2021-08-21T15:04:38.825 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'11 2021-08-21T15:04:38.825 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'12 2021-08-21T15:04:38.825 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'13 2021-08-21T15:04:38.825 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'14 2021-08-21T15:04:38.826 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'15 2021-08-21T15:04:38.826 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'16 2021-08-21T15:04:38.826 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'17 2021-08-21T15:04:38.826 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'18 2021-08-21T15:04:38.826 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'19 2021-08-21T15:04:38.827 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'20 2021-08-21T15:04:38.827 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'21 2021-08-21T15:04:38.827 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'22 2021-08-21T15:04:38.828 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'23 2021-08-21T15:04:38.828 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'24 2021-08-21T15:04:38.828 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'25 2021-08-21T15:04:38.828 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'26 2021-08-21T15:04:38.828 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'27 2021-08-21T15:04:38.829 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'28 2021-08-21T15:04:38.829 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'29 2021-08-21T15:04:38.829 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'30 2021-08-21T15:04:38.829 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'31 2021-08-21T15:04:38.830 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'32 2021-08-21T15:04:38.830 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'33 2021-08-21T15:04:38.830 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'34 2021-08-21T15:04:38.830 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'35 2021-08-21T15:04:38.830 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'36 2021-08-21T15:04:38.831 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'37 2021-08-21T15:04:38.831 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'38 2021-08-21T15:04:38.831 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'39 2021-08-21T15:04:38.831 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'40 2021-08-21T15:04:38.832 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'41 2021-08-21T15:04:38.832 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'42 2021-08-21T15:04:38.832 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'43 2021-08-21T15:04:38.832 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'44 2021-08-21T15:04:38.832 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'45 2021-08-21T15:04:38.833 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'46 2021-08-21T15:04:38.833 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'47 2021-08-21T15:04:38.833 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'48 2021-08-21T15:04:38.833 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'49 2021-08-21T15:04:38.834 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'50 2021-08-21T15:04:38.834 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'51 2021-08-21T15:04:38.834 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'52 2021-08-21T15:04:38.834 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'53 2021-08-21T15:04:38.834 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'54 2021-08-21T15:04:38.835 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'55 2021-08-21T15:04:38.835 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'56 2021-08-21T15:04:38.835 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'57 2021-08-21T15:04:38.835 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'58 2021-08-21T15:04:38.836 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'59 2021-08-21T15:04:38.836 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'60 2021-08-21T15:04:38.836 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'61 2021-08-21T15:04:38.836 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'62 2021-08-21T15:04:38.836 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'63 2021-08-21T15:04:38.837 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'64 2021-08-21T15:04:38.837 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'65 2021-08-21T15:04:38.837 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'66 2021-08-21T15:04:38.837 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'67 2021-08-21T15:04:38.837 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'68 2021-08-21T15:04:38.838 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'69 2021-08-21T15:04:38.838 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'70 2021-08-21T15:04:38.838 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'71 2021-08-21T15:04:38.838 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'72 2021-08-21T15:04:38.839 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'73 2021-08-21T15:04:38.839 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'74 2021-08-21T15:04:38.839 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'75 2021-08-21T15:04:38.839 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'76 2021-08-21T15:04:38.839 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'77 2021-08-21T15:04:38.840 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'78 2021-08-21T15:04:38.840 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'79 2021-08-21T15:04:38.840 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'80 2021-08-21T15:04:38.840 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'81 2021-08-21T15:04:38.841 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'82 2021-08-21T15:04:38.841 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'83 2021-08-21T15:04:38.841 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'84 2021-08-21T15:04:38.841 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'85 2021-08-21T15:04:38.841 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'86 2021-08-21T15:04:38.842 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'87 2021-08-21T15:04:38.842 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'88 2021-08-21T15:04:38.842 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'89 2021-08-21T15:04:38.842 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'90 2021-08-21T15:04:38.843 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'91 2021-08-21T15:04:38.843 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'92 2021-08-21T15:04:38.843 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'93 2021-08-21T15:04:38.843 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'94 2021-08-21T15:04:38.843 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'95 2021-08-21T15:04:38.845 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'96 2021-08-21T15:04:38.845 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'97 2021-08-21T15:04:38.845 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'98 2021-08-21T15:04:38.846 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'99 2021-08-21T15:04:38.846 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup version 34'100 2021-08-21T15:04:38.846 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'101 2021-08-21T15:04:38.846 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'102 2021-08-21T15:04:38.847 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'103 2021-08-21T15:04:38.847 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'104 2021-08-21T15:04:38.847 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'105 2021-08-21T15:04:38.847 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'106 2021-08-21T15:04:38.847 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'107 2021-08-21T15:04:38.848 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'108 2021-08-21T15:04:38.848 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'109 2021-08-21T15:04:38.849 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'110 2021-08-21T15:04:38.849 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'111 2021-08-21T15:04:38.849 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'112 2021-08-21T15:04:38.849 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'113 2021-08-21T15:04:38.849 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'114 2021-08-21T15:04:38.850 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'115 2021-08-21T15:04:38.850 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'116 2021-08-21T15:04:38.850 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'117 2021-08-21T15:04:38.850 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'118 2021-08-21T15:04:38.851 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'119 2021-08-21T15:04:38.851 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'120 2021-08-21T15:04:38.851 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'121 2021-08-21T15:04:38.851 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'122 2021-08-21T15:04:38.851 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'123 2021-08-21T15:04:38.852 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'124 2021-08-21T15:04:38.852 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'125 2021-08-21T15:04:38.853 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'126 2021-08-21T15:04:38.853 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'127 2021-08-21T15:04:38.853 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'128 2021-08-21T15:04:38.853 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'129 2021-08-21T15:04:38.854 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'130 2021-08-21T15:04:38.854 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'131 2021-08-21T15:04:38.854 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'132 2021-08-21T15:04:38.854 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'133 2021-08-21T15:04:38.854 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'134 2021-08-21T15:04:38.855 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'135 2021-08-21T15:04:38.855 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'136 2021-08-21T15:04:38.855 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'137 2021-08-21T15:04:38.855 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'138 2021-08-21T15:04:38.856 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'139 2021-08-21T15:04:38.856 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'140 2021-08-21T15:04:38.856 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'141 2021-08-21T15:04:38.856 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'142 2021-08-21T15:04:38.857 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'143 2021-08-21T15:04:38.857 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'144 2021-08-21T15:04:38.857 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'145 2021-08-21T15:04:38.857 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'146 2021-08-21T15:04:38.857 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'147 2021-08-21T15:04:38.858 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'148 2021-08-21T15:04:38.858 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'149 2021-08-21T15:04:38.858 INFO:tasks.workunit.client.0.smithi077.stdout:td/osd-backfill-recovery-log/osd.0.log:2021-08-21T15:04:28.050+0000 7fbe4c11e700 20 copy_up_to/copy_after copy dup from log version 34'150 2021-08-21T15:04:38.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2021-08-21T15:04:38.871 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2021-08-21T15:04:38.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2021-08-21T15:04:38.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:38.872 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T15:04:38.873 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-08-21T15:04:38.873 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T15:04:38.873 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T15:04:38.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T15:04:38.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T15:04:38.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T15:04:38.874 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T15:04:38.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T15:04:38.875 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T15:04:38.875 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T15:04:38.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T15:04:38.876 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T15:04:38.876 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T15:04:38.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T15:04:38.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T15:04:38.877 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T15:04:38.878 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T15:04:38.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T15:04:38.878 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T15:04:38.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-08-21T15:04:38.879 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T15:04:38.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:38.880 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:38.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53763 2021-08-21T15:04:38.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T15:04:38.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T15:04:38.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2021-08-21T15:04:38.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2021-08-21T15:04:38.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:38.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2021-08-21T15:04:38.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:38.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2021-08-21T15:04:38.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2021-08-21T15:04:38.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2021-08-21T15:04:38.883 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2021-08-21T15:04:38.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2021-08-21T15:04:38.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2021-08-21T15:04:38.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2021-08-21T15:04:38.884 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2021-08-21T15:04:38.885 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2021-08-21T15:04:38.885 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2021-08-21T15:04:38.885 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2021-08-21T15:04:38.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2021-08-21T15:04:38.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2021-08-21T15:04:38.886 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2021-08-21T15:04:38.886 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T15:04:38.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2021-08-21T15:04:38.887 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2021-08-21T15:04:38.887 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-08-21T15:04:38.888 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2021-08-21T15:04:38.888 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2021-08-21T15:04:38.888 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2021-08-21T15:04:38.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2021-08-21T15:04:38.889 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:38.890 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:38.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.53763 2021-08-21T15:04:38.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2021-08-21T15:04:38.890 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2021-08-21T15:04:38.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2021-08-21T15:04:38.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2021-08-21T15:04:38.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:38.891 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:38.891 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.53763 2021-08-21T15:04:38.892 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2021-08-21T15:04:38.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2021-08-21T15:04:38.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2021-08-21T15:04:38.892 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2021-08-21T15:04:38.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: TEST_recovery_1 td/osd-backfill-recovery-log 2021-08-21T15:04:38.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:121: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:38.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:123: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2021-08-21T15:04:38.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:38.893 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-08-21T15:04:38.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local loglen=40 2021-08-21T15:04:38.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=0 2021-08-21T15:04:38.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local objects=40 2021-08-21T15:04:38.894 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2021-08-21T15:04:38.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2021-08-21T15:04:38.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2021-08-21T15:04:38.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:38.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2021-08-21T15:04:38.895 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2021-08-21T15:04:38.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2021-08-21T15:04:38.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2021-08-21T15:04:38.896 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2021-08-21T15:04:39.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2021-08-21T15:04:39.147 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T15:04:39.148 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T15:04:39.148 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T15:04:39.148 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:39.149 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:39.149 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:04:39.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2021-08-21T15:04:39.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2021-08-21T15:04:39.409 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2021-08-21T15:04:39.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T15:04:39.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T15:04:39.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=fsid 2021-08-21T15:04:39.410 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .fsid 2021-08-21T15:04:39.411 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T15:04:39.411 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T15:04:39.411 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T15:04:39.411 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T15:04:39.412 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:39.412 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:39.412 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53763/ceph-mon.a.asok 2021-08-21T15:04:39.413 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T15:04:39.413 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.53763/ceph-mon.a.asok config get fsid 2021-08-21T15:04:39.506 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2021-08-21T15:04:39.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: get_config: local daemon=mon 2021-08-21T15:04:39.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: get_config: local id=a 2021-08-21T15:04:39.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: get_config: local config=mon_host 2021-08-21T15:04:39.507 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1110: get_config: jq -r .mon_host 2021-08-21T15:04:39.508 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: get_asok_path mon.a 2021-08-21T15:04:39.508 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2021-08-21T15:04:39.508 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2021-08-21T15:04:39.508 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2021-08-21T15:04:39.509 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:39.510 INFO:tasks.workunit.client.0.smithi077.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:39.510 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.53763/ceph-mon.a.asok 2021-08-21T15:04:39.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: CEPH_ARGS= 2021-08-21T15:04:39.510 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: get_config: ceph --format json daemon /tmp/ceph-asok.53763/ceph-mon.a.asok config get mon_host 2021-08-21T15:04:39.602 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2021-08-21T15:04:39.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:39.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2021-08-21T15:04:39.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2021-08-21T15:04:39.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2021-08-21T15:04:39.603 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-backfill-recovery-log/x 2021-08-21T15:04:39.604 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2021-08-21T15:04:40.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2021-08-21T15:04:40.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T15:04:40.489 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T15:04:40.490 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T15:04:40.490 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:40.491 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:40.492 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:04:40.492 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T15:04:40.492 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2021-08-21T15:04:40.513 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2021-08-21T15:04:40.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-08-21T15:04:40.514 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-08-21T15:04:40.515 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2021-08-21T15:04:40.521 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2021-08-21T15:04:40.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T15:04:40.521 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2021-08-21T15:04:40.522 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T15:04:40.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T15:04:40.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2021-08-21T15:04:40.523 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T15:04:40.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2021-08-21T15:04:40.524 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T15:04:40.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T15:04:40.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T15:04:40.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T15:04:40.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2021-08-21T15:04:40.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2021-08-21T15:04:40.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T15:04:40.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-08-21T15:04:40.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T15:04:40.531 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T15:04:40.531 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T15:04:40.531 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T15:04:40.532 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T15:04:40.533 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:04:40.533 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:04:40.534 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:04:40.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:04:40.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T15:04:40.535 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T15:04:40.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T15:04:40.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T15:04:40.536 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T15:04:40.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T15:04:40.537 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T15:04:40.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T15:04:40.539 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T15:04:40.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T15:04:40.540 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2021-08-21T15:04:40.541 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T15:04:40.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=41d5ec58-f543-48b3-80b0-6ec14c6e216d 2021-08-21T15:04:40.542 INFO:tasks.workunit.client.0.smithi077.stdout:add osd0 41d5ec58-f543-48b3-80b0-6ec14c6e216d 2021-08-21T15:04:40.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd0 41d5ec58-f543-48b3-80b0-6ec14c6e216d' 2021-08-21T15:04:40.545 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T15:04:40.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAIFiFhOq3CHxAAvxxaESsv8bMVbr/D9nQr1A== 2021-08-21T15:04:40.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAIFiFhOq3CHxAAvxxaESsv8bMVbr/D9nQr1A=="}' 2021-08-21T15:04:40.546 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new 41d5ec58-f543-48b3-80b0-6ec14c6e216d -i td/osd-backfill-recovery-log/0/new.json 2021-08-21T15:04:40.891 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T15:04:40.902 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2021-08-21T15:04:40.903 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 0 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAIFiFhOq3CHxAAvxxaESsv8bMVbr/D9nQr1A== --osd-uuid 41d5ec58-f543-48b3-80b0-6ec14c6e216d 2021-08-21T15:04:40.929 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:40.928+0000 7fafe3b260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:04:40.930 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:40.930+0000 7fafe3b260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:04:40.931 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:40.931+0000 7fafe3b260c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:04:40.932 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:40.931+0000 7fafe3b260c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T15:04:40.932 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:40.931+0000 7fafe3b260c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T15:04:40.933 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:40.931+0000 7fafe3b260c0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2021-08-21T15:04:41.007 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:41.006+0000 7fafe3b260c0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2021-08-21T15:04:46.731 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2021-08-21T15:04:46.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T15:04:46.732 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd0 key to auth repository 2021-08-21T15:04:46.733 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd0 key to auth repository 2021-08-21T15:04:46.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T15:04:47.242 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.0 2021-08-21T15:04:47.243 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.0 2021-08-21T15:04:47.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 0 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T15:04:47.244 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T15:04:47.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T15:04:47.245 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T15:04:47.266 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:47.265+0000 7f627bc6f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:04:47.267 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:47.267+0000 7f627bc6f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:04:47.268 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:47.268+0000 7f627bc6f0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:04:47.650 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 0 2021-08-21T15:04:47.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T15:04:47.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=0 2021-08-21T15:04:47.651 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T15:04:47.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T15:04:47.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:47.652 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T15:04:47.653 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T15:04:47.653 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:47.654 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:47.792 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:47.792+0000 7f627bc6f0c0 -1 Falling back to public interface 2021-08-21T15:04:48.026 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:49.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:49.028 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:49.028 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T15:04:49.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T15:04:49.029 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:49.030 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:49.453 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:50.186 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:04:50.180+0000 7f627bc6f0c0 -1 osd.0 0 log_to_monitors {default=true} 2021-08-21T15:04:50.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:50.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:50.454 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T15:04:50.455 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T15:04:50.455 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:50.456 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:50.858 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:51.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:51.860 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:51.860 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T15:04:51.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T15:04:51.861 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:51.862 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:52.294 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:53.296 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:53.297 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:53.297 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T15:04:53.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T15:04:53.298 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:53.299 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:53.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:54.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:54.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:54.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T15:04:54.961 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T15:04:54.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:54.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:55.148 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:56.149 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:56.150 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:56.150 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T15:04:56.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T15:04:56.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:56.151 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:56.584 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:57.586 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:57.587 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:57.587 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T15:04:57.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T15:04:57.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:57.589 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:57.994 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:04:58.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:04:58.995 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:04:58.996 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T15:04:58.996 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T15:04:58.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:04:58.997 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:04:59.376 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:00.377 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:00.378 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:00.378 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T15:05:00.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T15:05:00.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:00.379 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:00.879 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:01.880 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:01.881 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:01.882 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T15:05:01.882 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T15:05:01.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:01.883 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:02.403 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:03.404 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:03.405 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:03.405 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T15:05:03.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T15:05:03.406 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:03.407 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:03.823 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:04.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:04.824 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:04.825 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T15:05:04.826 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T15:05:04.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:04.827 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:05.334 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:06.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:06.335 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:06.336 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T15:05:06.336 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T15:05:06.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:06.337 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:06.738 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:07.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:07.743 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:07.744 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T15:05:07.744 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T15:05:07.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:07.745 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:08.155 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:09.157 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:09.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:09.158 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T15:05:09.159 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T15:05:09.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:09.159 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:09.803 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:10.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:10.807 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:10.808 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T15:05:10.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T15:05:10.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:10.809 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:11.541 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:12.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:12.543 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:12.544 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T15:05:12.544 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T15:05:12.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:12.545 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.0 up' 2021-08-21T15:05:12.951 INFO:tasks.workunit.client.0.smithi077.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/116508,v1:127.0.0.1:6803/116508] [v2:127.0.0.1:6804/116508,v1:127.0.0.1:6805/116508] exists,up 41d5ec58-f543-48b3-80b0-6ec14c6e216d 2021-08-21T15:05:12.955 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T15:05:12.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T15:05:12.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T15:05:12.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T15:05:12.956 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2021-08-21T15:05:12.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T15:05:12.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T15:05:12.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2021-08-21T15:05:12.957 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T15:05:12.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2021-08-21T15:05:12.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T15:05:12.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T15:05:12.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T15:05:12.958 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T15:05:12.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2021-08-21T15:05:12.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2021-08-21T15:05:12.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T15:05:12.959 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-08-21T15:05:12.960 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T15:05:12.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T15:05:12.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T15:05:12.960 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T15:05:12.960 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T15:05:12.961 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:05:12.961 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:05:12.961 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:05:12.961 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:05:12.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T15:05:12.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T15:05:12.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T15:05:12.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T15:05:12.962 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T15:05:12.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T15:05:12.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T15:05:12.963 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T15:05:12.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T15:05:12.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T15:05:12.965 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2021-08-21T15:05:12.966 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T15:05:12.966 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=c4485ca3-5c38-4abe-9794-50d5db2d7188 2021-08-21T15:05:12.967 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd1 c4485ca3-5c38-4abe-9794-50d5db2d7188' 2021-08-21T15:05:12.967 INFO:tasks.workunit.client.0.smithi077.stdout:add osd1 c4485ca3-5c38-4abe-9794-50d5db2d7188 2021-08-21T15:05:12.968 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T15:05:12.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQAoFiFhc1RnOhAAgdbadIGQe+pxKyZdmjRdnA== 2021-08-21T15:05:12.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQAoFiFhc1RnOhAAgdbadIGQe+pxKyZdmjRdnA=="}' 2021-08-21T15:05:12.980 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new c4485ca3-5c38-4abe-9794-50d5db2d7188 -i td/osd-backfill-recovery-log/1/new.json 2021-08-21T15:05:13.476 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T15:05:13.483 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2021-08-21T15:05:13.484 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 1 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAoFiFhc1RnOhAAgdbadIGQe+pxKyZdmjRdnA== --osd-uuid c4485ca3-5c38-4abe-9794-50d5db2d7188 2021-08-21T15:05:13.509 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.508+0000 7f50bc2a90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:13.510 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.510+0000 7f50bc2a90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.510+0000 7f50bc2a90c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.511+0000 7f50bc2a90c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T15:05:13.512 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.511+0000 7f50bc2a90c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T15:05:13.518 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.511+0000 7f50bc2a90c0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2021-08-21T15:05:13.564 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:13.560+0000 7f50bc2a90c0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2021-08-21T15:05:19.293 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2021-08-21T15:05:19.503 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T15:05:19.504 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd1 key to auth repository 2021-08-21T15:05:19.505 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd1 key to auth repository 2021-08-21T15:05:19.505 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T15:05:19.788 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.1 2021-08-21T15:05:19.789 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.1 2021-08-21T15:05:19.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 1 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T15:05:19.790 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T15:05:19.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T15:05:19.791 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T15:05:19.816 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:19.813+0000 7fdd5c63b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:19.816 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:19.815+0000 7fdd5c63b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:19.817 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:19.815+0000 7fdd5c63b0c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:20.176 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 1 2021-08-21T15:05:20.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T15:05:20.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=1 2021-08-21T15:05:20.177 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T15:05:20.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T15:05:20.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:20.178 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T15:05:20.178 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T15:05:20.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:20.179 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:20.335 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:20.332+0000 7fdd5c63b0c0 -1 Falling back to public interface 2021-08-21T15:05:20.617 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:21.619 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:21.620 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:21.622 INFO:tasks.workunit.client.0.smithi077.stdout:1 2021-08-21T15:05:21.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 1 2021-08-21T15:05:21.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:21.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:22.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:22.713 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:22.712+0000 7fdd5c63b0c0 -1 osd.1 0 log_to_monitors {default=true} 2021-08-21T15:05:23.005 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:23.008 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:23.009 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T15:05:23.009 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 2 2021-08-21T15:05:23.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:23.010 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:23.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:24.624 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:24.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:24.625 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 3 2021-08-21T15:05:24.625 INFO:tasks.workunit.client.0.smithi077.stdout:3 2021-08-21T15:05:24.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:24.626 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:25.052 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:26.053 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:26.054 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:26.054 INFO:tasks.workunit.client.0.smithi077.stdout:4 2021-08-21T15:05:26.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 4 2021-08-21T15:05:26.055 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:26.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:26.517 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:27.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:27.518 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:27.519 INFO:tasks.workunit.client.0.smithi077.stdout:5 2021-08-21T15:05:27.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 5 2021-08-21T15:05:27.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:27.520 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:28.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:29.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:29.038 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:29.047 INFO:tasks.workunit.client.0.smithi077.stdout:6 2021-08-21T15:05:29.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 6 2021-08-21T15:05:29.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:29.048 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:29.426 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:30.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:30.423 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:30.424 INFO:tasks.workunit.client.0.smithi077.stdout:7 2021-08-21T15:05:30.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 7 2021-08-21T15:05:30.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:30.425 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:30.839 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:31.840 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:31.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:31.841 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 8 2021-08-21T15:05:31.842 INFO:tasks.workunit.client.0.smithi077.stdout:8 2021-08-21T15:05:31.842 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:31.843 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:32.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:33.321 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:33.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:33.322 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 9 2021-08-21T15:05:33.323 INFO:tasks.workunit.client.0.smithi077.stdout:9 2021-08-21T15:05:33.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:33.323 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:33.733 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:34.734 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:34.735 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:34.735 INFO:tasks.workunit.client.0.smithi077.stdout:10 2021-08-21T15:05:34.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 10 2021-08-21T15:05:34.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:34.736 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:35.313 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:36.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:36.314 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:36.315 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 11 2021-08-21T15:05:36.315 INFO:tasks.workunit.client.0.smithi077.stdout:11 2021-08-21T15:05:36.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:36.316 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:37.434 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:38.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:38.435 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:38.436 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 12 2021-08-21T15:05:38.436 INFO:tasks.workunit.client.0.smithi077.stdout:12 2021-08-21T15:05:38.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:38.437 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:38.828 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:39.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:39.830 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:39.830 INFO:tasks.workunit.client.0.smithi077.stdout:13 2021-08-21T15:05:39.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 13 2021-08-21T15:05:39.831 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:39.832 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:40.281 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:41.283 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:41.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:41.303 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 14 2021-08-21T15:05:41.304 INFO:tasks.workunit.client.0.smithi077.stdout:14 2021-08-21T15:05:41.304 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:41.305 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:41.680 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:42.699 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:42.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:42.700 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 15 2021-08-21T15:05:42.701 INFO:tasks.workunit.client.0.smithi077.stdout:15 2021-08-21T15:05:42.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:42.701 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:43.679 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:44.669 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:44.671 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:44.671 INFO:tasks.workunit.client.0.smithi077.stdout:16 2021-08-21T15:05:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 16 2021-08-21T15:05:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:44.672 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:45.118 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: wait_for_osd: sleep 1 2021-08-21T15:05:46.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i++ )) 2021-08-21T15:05:46.112 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:46.113 INFO:tasks.workunit.client.0.smithi077.stdout:17 2021-08-21T15:05:46.113 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 17 2021-08-21T15:05:46.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:46.114 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.1 up' 2021-08-21T15:05:46.500 INFO:tasks.workunit.client.0.smithi077.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/118352,v1:127.0.0.1:6811/118352] [v2:127.0.0.1:6812/118352,v1:127.0.0.1:6813/118352] exists,up c4485ca3-5c38-4abe-9794-50d5db2d7188 2021-08-21T15:05:46.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: wait_for_osd: status=0 2021-08-21T15:05:46.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:966: wait_for_osd: break 2021-08-21T15:05:46.525 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: return 0 2021-08-21T15:05:46.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2021-08-21T15:05:46.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2021-08-21T15:05:46.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-backfill-recovery-log 2021-08-21T15:05:46.526 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2021-08-21T15:05:46.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2021-08-21T15:05:46.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2021-08-21T15:05:46.527 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2021-08-21T15:05:46.528 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2021-08-21T15:05:46.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2021-08-21T15:05:46.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2021-08-21T15:05:46.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2021-08-21T15:05:46.529 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2021-08-21T15:05:46.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2021-08-21T15:05:46.530 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2021-08-21T15:05:46.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2021-08-21T15:05:46.531 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2021-08-21T15:05:46.531 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2021-08-21T15:05:46.531 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2021-08-21T15:05:46.532 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2021-08-21T15:05:46.533 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2021-08-21T15:05:46.533 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2021-08-21T15:05:46.533 INFO:tasks.workunit.client.0.smithi077.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.53763 2021-08-21T15:05:46.533 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:05:46.534 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' 2021-08-21T15:05:46.566 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2021-08-21T15:05:46.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2021-08-21T15:05:46.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2021-08-21T15:05:46.567 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2021-08-21T15:05:46.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2021-08-21T15:05:46.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2021-08-21T15:05:46.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2021-08-21T15:05:46.568 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T15:05:46.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' ' 2021-08-21T15:05:46.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+= 2021-08-21T15:05:46.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2021-08-21T15:05:46.569 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: uuidgen 2021-08-21T15:05:46.569 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local uuid=a100cc1e-8ae7-456f-8394-0b1124032e8a 2021-08-21T15:05:46.570 INFO:tasks.workunit.client.0.smithi077.stdout:add osd2 a100cc1e-8ae7-456f-8394-0b1124032e8a 2021-08-21T15:05:46.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo 'add osd2 a100cc1e-8ae7-456f-8394-0b1124032e8a' 2021-08-21T15:05:46.580 INFO:tasks.workunit.client.0.smithi077.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph-authtool --gen-print-key 2021-08-21T15:05:46.580 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: OSD_SECRET=AQBKFiFhE4uXHhAADSTqc4IBRUhKJhFpr1SJVw== 2021-08-21T15:05:46.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: echo '{"cephx_secret": "AQBKFiFhE4uXHhAADSTqc4IBRUhKJhFpr1SJVw=="}' 2021-08-21T15:05:46.581 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph osd new a100cc1e-8ae7-456f-8394-0b1124032e8a -i td/osd-backfill-recovery-log/2/new.json 2021-08-21T15:05:47.048 INFO:tasks.workunit.client.0.smithi077.stdout:2 2021-08-21T15:05:47.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2021-08-21T15:05:47.056 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph-osd -i 2 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBKFiFhE4uXHhAADSTqc4IBRUhKJhFpr1SJVw== --osd-uuid a100cc1e-8ae7-456f-8394-0b1124032e8a 2021-08-21T15:05:47.080 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.079+0000 7fc07fcc20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:47.081 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.080+0000 7fc07fcc20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:47.082 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.081+0000 7fc07fcc20c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:47.082 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.081+0000 7fc07fcc20c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T15:05:47.082 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.081+0000 7fc07fcc20c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T15:05:47.083 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.081+0000 7fc07fcc20c0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2021-08-21T15:05:47.151 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:47.151+0000 7fc07fcc20c0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2021-08-21T15:05:53.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2021-08-21T15:05:53.130 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: cat 2021-08-21T15:05:53.132 INFO:tasks.workunit.client.0.smithi077.stdout:adding osd2 key to auth repository 2021-08-21T15:05:53.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo adding osd2 key to auth repository 2021-08-21T15:05:53.133 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2021-08-21T15:05:53.634 INFO:tasks.workunit.client.0.smithi077.stdout:start osd.2 2021-08-21T15:05:53.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: echo start osd.2 2021-08-21T15:05:53.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: ceph-osd -i 2 --fsid=b2863fef-04af-4d5f-b2ab-407342110d7c --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53763/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2021-08-21T15:05:53.636 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: ceph osd dump --format=json 2021-08-21T15:05:53.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: jq '.flags_set[]' 2021-08-21T15:05:53.637 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:680: run_osd: grep -q '"noup"' 2021-08-21T15:05:53.666 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:53.657+0000 7f55dc4610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:53.667 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:53.659+0000 7f55dc4610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:53.667 INFO:tasks.workunit.client.0.smithi077.stderr:2021-08-21T15:05:53.660+0000 7f55dc4610c0 -1 WARNING: all dangerous and experimental features are enabled. 2021-08-21T15:05:54.033 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:683: run_osd: wait_for_osd up 2 2021-08-21T15:05:54.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: wait_for_osd: local state=up 2021-08-21T15:05:54.034 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:957: wait_for_osd: local id=2 2021-08-21T15:05:54.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: wait_for_osd: status=1 2021-08-21T15:05:54.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i=0 )) 2021-08-21T15:05:54.035 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: wait_for_osd: (( i < 300 )) 2021-08-21T15:05:54.036 INFO:tasks.workunit.client.0.smithi077.stdout:0 2021-08-21T15:05:54.037 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:961: wait_for_osd: echo 0 2021-08-21T15:05:54.057 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: ceph osd dump 2021-08-21T15:05:54.058 INFO:tasks.workunit.client.0.smithi077.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: wait_for_osd: grep 'osd.2 up' 2021-08-21T15:05:54.178 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-08-21T15:05:54.205 DEBUG:teuthology.task.console_log:Killing console logger for smithi077 2021-08-21T15:05:54.209 DEBUG:teuthology.exit:Finished running handlers